Project planning meeting - 21 July 2006
(Bought forward from the planned date of 25 Jul, as we seem to have reached a point of having several things to discuss.)
Present: Stuart Yeates (SY), Graham Klyne (GK)
Last report: SakaiVre/PlanningProgress/20060705
This report: SakaiVre/PlanningProgress/20060721
Next meeting: Monday 7 August 2006, 09:00. SakaiVre/PlanningProgress/20060807
- Activity since last report
- Notes for next meeting
- Review agenda
- Confirm date/time of next meeting
- Discussion of specific issues
- Progress on Shibboleth/Sakai integration
- Interaction with Sakai community
- Guan Xi
- Progressing coordination of Shibboleth roll-out
- Progressing Shared Shibboleth facilities in Oxford
- Sakai system performance problems
- Progress on Shibboleth/Sakai integration
- Review actions outstanding at last meeting
- New activities and issues arising from current work
2. Activity since last report
Our main focus has been to integrate Shibboleth-provided authentication with Sakai. We have succeeded in getting Sakai to recognize a Shibboleth-provided username ("Level 0" according to SakaiVre/ShibbolethWebAuthIntegration#Levels), but are still struggling to understand how to Sakai can use Shibboleth attributes for use account creation and determination of role membership. Thus, the continuimg immediate focus of our activities is in action 20060619.3.
2.1. Actions closed or completed
[GK] add details from David Spence (CCLRC/RAL) to wiki page at SakaiVre/ShibbolethFederation, and email Sakai project list in attempt to prompt further responses. Done. See http://tools.iso.port.ac.uk/pipermail/sakai-vre/2006-July/000068.html.
[SY] draft a paragraph about LDAP/Eduserve groups to add to the federation page at SakaiVre/ShibbolethFederation (or some other appropriate location). Done - see http://maillist.ox.ac.uk/ezmlm-cgi?3855:mss:96:200607:bjpidnkgfknbdhbakack. Comments have been added to SakaiVre/AttributeMappingsTable.
[GK] Add note aboput EduPerson attributes (cf. EduCause site) to the federation page at SakaiVre/ShibbolethFederation. Done.
[SY] Send URI link to EduPerson attribute information to local project mailing list (in support of action 20060705.4). Done. See SakaiVre/ShibbolethFederation.
[GK] Review RDF from Stuart's conversion of LDAP user information, and provide feedback. Done - see http://maillist.ox.ac.uk/ezmlm-cgi?3855:mss:118:200607:ikemnppigeipmjkjpgcf.
[GK] Add Shibboleth authentication to Sakai. We are now ready to start looking into this. Done (in a limited sense). See SakaiVre/SakaiUserDirectoryProvider, section 3. Further development will be recorded under action 20060619.3.
[GK] Investigate Sakai background technologies (Spring, JSF) (See SakaiNotes; TODO: input concerning JSF.) Closed. The JSF elements of Sakai are increasingly not looking relevant to our activity, so we've decided to close this part of the activity as incomplete. But I did recently notice more material about Sakai's use of Spring dependency injection here: https://source.sakaiproject.org/svn/reference/trunk/docs/architecture/sakai_component.doc.
Documented use-cases for Shibboleth-enabling of Sakai, in response to questions from Sakai developers. See SakaiVre/ShibbolethSakaiUseCases.
Discovered some Sakai documents of possible interest to us: http://maillist.ox.ac.uk/ezmlm-cgi?3855:mss:117:200607:gccajhpejmaikahjdmgb
2.2. Actions progressed
[GK/SY] Initiated investigation of how to make Shibboleth attributes available for use in Sakai access controls. Initial investigation of Sakai web site is not very helpful. Investigations are being recorded at SakaiVre/SakaiUserDirectoryProvider. See discussion.
- [SY] Investigate performance problems with our Sakai/Shibboleth demo machine, and propose appropriate upgrade or route resolution. Ongoing - see discussion.
2.3. New activities and notes
- [SY+GK] Arrange luchtime meeting with Adam Marshall to explore any possible relevance of the Guan Xi work to our goals.
- [SY+GK] Arrange meeting with Christian Fernau to further discuss options for combining our Shibbboleth IdP facilities.
- [SY+GK] Modify and document Sakai configuration to persist user database. (Currently, new users added to Sakai are lost on each restart of the system.)
- [SY+GK] Explore use of Sakai's authorization framework, to get a better understanding of the way it employs the user, role and permission concepts, and how they may interact with external authentication and authorization.
- [SY,GK] Explore Sakai architectural documents to look for clues about user/role provider functions ("provider" being the Sakai term for components that link to external information sources).
- [GK] Revisit Sakai "provider" source code to see if it makes more sense in light of what we now know about Sakai container authentication. Formulate tentative integration plan to be posed as a "could this work?" style of question to the Sakai community.
- [GK] Announce progress on Sakai authentication to project partners, and repeat request for information about Shibboleth and/or institutional authentication/directory deployments, and how to progress a project-wide deployment.
[SY] Survey recent Tetra-ELF mailing list archives to see if there are any overlooked messages there that are helpful to our cause. (See http://sourceforge.net/mailarchive/forum.php?forum=tetraelf-developers.)
2.4. Summary of ongoing actions brought forward
- [SY] Investigate OUCS procedures for spinning up new shared trial facilities.
- [SY] Analysis of search requirements. (On hold while we focus our efforts on the Shibboleth/Sakai integration.)
[GK] Port SPIE Shibboleth/WSRP (cf. work by Jasper Tredgold) to Sakai: (Waiting for 20060301.10) Install Shibboleth/WSRP software locally, and convert to work with Sakai. The main remaining unknown is to get Shibboleth attributes into the Sakai portal framework - this has been separated out as a new action 20060619.3. No further progress. The value of WSRP is cast further into doubt by this message from Charles Severance: http://maillist.ox.ac.uk/ezmlm-cgi?3855:mss:119:200607:nelfnlficidkopiknchj.
[GK] Initiate process of SDSS federation credentials. Have obtained an SDSS CA certificate, but have not yet joined the SDSS federation. See http://maillist.ox.ac.uk/ezmlm-cgi?3855:mss:84:200606:hbmpafckdeclpnjmmgmb. May be overtaken by events with respect to action 20060721.2 (see discussion).
[GK/SY] Initiated investigation of how to make Shibboleth attributes available for use in Sakai access controls. Initial investigation of Sakai web site is not very helpful. Investighations are being recorded at SakaiVre/SakaiUserDirectoryProvider
- [GK/SY] Write Shibboleth configuration document, in particular showing how the various configuration elements and functions are inter-related.
- [SY] Investigate performance problems with our Sakai demo machine. The problem appears to be memory shortage, and the present machine cannot be usefully upgraded, so we're now casting about for a new system, mainly with greater RAM memory capacity or potential capacity.
3.1. Shibboleth integration with Sakai
We have succeeded in getting Sakai to recognize a Shibboleth-provided username ("Level 0" according to SakaiVre/ShibbolethWebAuthIntegration#Levels), but are still struggling to understand how to Sakai can use Shibboleth attributes for use account creation and determination of role membership.
Sakai is proving to be a big and complex piece of software, and it is not always easy to navigate the source code as the Dependency Injection pattern (see SakaiNotes) makes it difficult to follow through code execution paths without wider configuration knowledge. This is impacting our ability to fully understand how Sakai can gather and use external attributes for role memberships, hence access control determination. There appears to be a goodly amount of high level architectural documentation, and of course there is the source code, but there's not much that links them together. So we seem to be very dependent on the community of Sakai users and developers to learn how to achieve integration with Shibboleth.
Options considered for "linking generalizations to code" in this area include:
- continue trawling the design documents and code base - but this is not efficient use of time
- continue probing the Sakai community - currently looking promising (see below)
- engage the CARET developers in discussion based on their experience with Sakai authentication systems
- pose a more specific question to our partners in the Sakai VRE project who have more knowledge of Sakai code
We've also been considering asking for a meeting or more detailed exchange of ideas with the CARET team in Cambridge, who appear to have addressed some similar problems with Sakai. We have some ongoing email exchanges but, pursuing a somewhat opportunistic tactic, are holding off escalating the level of attention requested until we have exhausted current lines of enquiry.
A possibile line of development for us to consider, once we better understand the Sakai integration issues, is to implement JAAS provider modules for Sakai, and then use the SPIE Shibboleth JAAS module to feed in values from Shibboleth. This is maybe more complex than a direct Shibboleth approach, but allows us to exploit available in-house expertise, and produces a more generic solution for Sakai. This is maybe an approach that might usefully be explored with the Sakai community, and may obtain further traction with them.
3.1.1. Interaction with Sakai community
Our efforts have recently attracted the attention of some Sakai developers, including Charles Severance, leading to our documenting use-cases for Sakai/Shibboleth integration (see SakaiVre/ShibbolethSakaiUseCases). The responses obtained so far have been of a rather rather hand-wavy "we can do this or that" kind, and not sufficient to direct our further efforts. But there's a clear awareness by some in the Sakai community that Shibboleth is important and needs to be accommodated, somehow. So if we can articulate the right questions, we may get some useful responses.
Currently, Sakai developers are engaged in related discussion on the Tetra-ELF mailing list, copying GK+SY, so that's a useful point of contact to develop.
Roughly, what we need to find out is this: How does Sakai get external attributes that are used to define user creation and role membership?
Some other specific questions:
- Is there a secific "howto" document for creating a user/role provider module?
- Shibboleth gives us a "principle user identifer" alocated by the home institute, and a hashed "Name Identifier" which is automaticallygenerated as a federation-wide unique identifier. How should these be mapped into Sakai concepts (such as ID and EID)?
3.1.2. Guan Xi
There have been suggestions that Guan Xi might be an alternative approach for what we're trying to achieve, so we'll try to organize a meeting with Adam Marshall to learn more about what Guan Xi is and what it can do.
3.2. Coordinating with other project partners
Currently available information about potential Shibboleth environments at all partner sites is summarized at SakaiVre/ShibbolethFederation. A section on existing Shibboleth IdP deployments has been added, as at least one other site (Daresbury CCLRC) has an existing deployment that probably can be used.
Progress on planning the roll-out has somewhat stalled. We have limited information from Lancaster and no information as yet from Portsmouth/Reading (though Mark Baker has acknowledged our request for information; see http://tools.iso.port.ac.uk/pipermail/sakai-vre/2006-July/000071.html).
It's not obvious to us how to drive this forward, or indeed who will be expected to perform installation and configuration at each site. Our next step will be to announce our recent progress on the main project mailing list, and repeat the request for engagement. Maybe this is an area where Mike Fraser might engage project leaders at other sites to help build some momentum for this activity?
3.3. Shared Shibboleth facilities in Oxford
Christian Fernau reports that the SPIE system has been moved from BLM to Xen virtual systems. Xen appears to be more stable and better supported, but with support for a limited number of Linux distributions. We might want to consider asking Christian for a Sandbox virtual machine on this Xen platform.
We also need to progress discussions with Christian about using a shared Shibboleth IdP for the Sakai project deployment. To this end, we need to ensure that we can serve out all the appropriate attributes from the SPIE IdP.
3.4. Sakai system performance problems
It looks as if the main problem is insufficient memory, leading to thrashing when Sakai is loaded. We have investigated adding memory to our present machine, but it's already maxed. Stuart has asked the inhouse team if there are any possible machines gathering dust in a cupboard - nothing matching our requirements.
Next steps (a) ask LTG if they have any spares, and (b) EITHER source a new machine OR chip in with Christian Fernau's Xen virtual machines http://www.cl.cam.ac.uk/Research/SRG/netos/xen/, buying some extra memory for it. (Budget should be available for this as so far, AFAICT, we haven't so far bought in any new hardware for this project.)
4. Notes for next meeting
(Matters arising following the planning meeting.)
[SY+GK] Arrange luchtime meeting with Adam Marshall to explore any possible relevance of the Gian Xi work to our goals. Done. A brief meeting revealed that there was not really any relevant experience in Oxford to draw upon here. Roughly, Guan Xi is an alternative implementation of Shibboleth with an e-learning bias. Other groups are looking at implementing Guian Xi for Sakai, so it may be that they'll solve the problems we are facing, as long as the Guan Xi SP module is interoperable with the Ineternet2 IdP module. I think there's little to be gained right now in us trying to learn about Guan Xi details, but we should monitor the status of their Sakai work. The person to be in touch with is Alistair Young at UHI. For more background, see http://www.guanxi.uhi.ac.uk/images/2/29/JISC_London_180706.pdf and http://www.guanxi.uhi.ac.uk/.
[SY+GK] Arrange meeting with Christian Fernau to further discuss options for combining out Shibbboleth IdP facilities. Done. Using the SPIE IdP should be quite easy for us. Points to consider include:
- We would need to agree on attributes to be released.
- SPIE IdP is already part of SDSS federation (among others). Our Sakai SP will also need to be registered with SDSS. Also, we can use one of Christian's sandbox VMs. For reliable Java operation, we would need to use a recognized Linux distro; mainly, the kernel must be a specific one tailored for Xen, and the CLIB must also have the correct optimization options. Debian looks most suitable here. We would need to provide a system image and a machine room IP address.
- Creating image: unix 'dump' files; create empty disk image file of desired size (e.g. using dd); use loopback connector to present file as block device; build system files on block device.
[SY,GK] Explore Sakai architectural documents to look for clues about user/role provider functions ("provider" being the Sakai term for components that link to external information sources). Done. Broader investigation continues.
[GK] Revisit Sakai "provider" source code to see if it makes more sense in light of what we now know about Sakai container authentication. Formulate tentative integration plan to be posed as a "could this work?" style of question to the Sakai community. Done. See http://maillist.ox.ac.uk/ezmlm-cgi?3855:mss:134:200607:gddpkpmonkgpgmkebbab. Since posting this message, there has been some lively discussion on the sakai-dev mailing list, and many details have becomne clearer. In particular, it seems that the Sakai authorization structure is a poor fit with Shibboleth attribute delivery. (New action to write up all this?) Alistair Young has encountered similar issues with his Guanxi development, and is taking the approach of constructing a "pod" of data from Shibboleth assertions, suitable for responding to Sakai authorization queries.
[GK] Announce progress on Sakai authentication to project partners, and repeat request for information about Shibboleth and/or institutional authentication/directory deployments, and how to progress a project-wide deployment. Done. See http://tools.iso.port.ac.uk/pipermail/sakai-vre/2006-July/000079.html. Our plan now will be to coordinate initially with just one partner - CCLRC?
We have started migrating Our Sakai installation to a new, faster machine, a Xen virtual Linux system hosted on a machine used by the SPIE project, running Debian Linux. The change to Debian is throwing up a few issues of system configuration that we need to work through. We'll keep the old machine running for a while as a Shibboleth test machine. See //wiki.oss-watch.ac.uk/DebianNotes.
-- GrahamKlyne 2006-07-21 08:34:06