Symfony Form widgets for many-to-many relationship: a new widget implementation - part II

Following part I, here is the final step to code a new symfony form widget in order to improve a many-to-many relationship. After discussing the best UI possible for a many-to-many relationship, we are now going to look how to code it as a symfony widget. I would not pretend it is the best solution, but I believe it is clean enough to be used and it definitively works!

One of the comment point me to the following approach: http://www.ryancramer.com/journal/entries/select_multiple/. It is also an elegant way to do it, unfortunately it does not work for me since my list of available object is extremely long, therefore a select is again not a good approach. Don't get me wrong, my approach is not necessarily the best but it fits a specific user case.

Before starting you can have a look at a demo of what we will get at the end. Here is a video to show the different interactions.

Pre-requisite:

• symfony 1.3/1.4
• jQuery 1.4+ and jQuery UI 1.8+. This might seems overkilling for a simple widget but most of the time you are using these 2 libraries in your project already. In jQuery UI you specifically need for this widget the new autocomplete widget.

Ok, no more talk, let's go to the code! You can find it as a gist at http://gist.github.com/443124. Feel free to clone, improve and share! This code need some refactoring for sure but I did not have time yet and many people where asking for it. Feel free to improve it and share it ;-)

Stay tuned, we will publish some new widgets in the coming months...

Symfony Form widget for many-to-many relationship: UI considerations - part I

This 2 parts tutorial will show you how we came up with a new symfony sfForm widget to handle many-to-many relationships. In our specific case, existing widgets for such relationships were found extremely unuser-friendly to say the least, we had to think about a new widget to help our users in their daily operation.
We do not pretend that our solution is the best in all user-cases but it is definitively a better approach when you have a lot of potential items to associate with your current object and a short list of associated items; the relationship between a classroom and students for example: you have a class of students (the class is less than 40 students usually) within a University of thousands of students.

The default widget

I have always been very skeptical with the default widget in symfony form framework for many-to-many relationship. In a UI perspective it was really bad:

• You have to "shift-click" in a multiple select box the item you want to add to your relation,
• if you have a long list, it is very inconvenient to find the one you want,
• if by mistake you click without the shift button pressed, you are good to start over your multiple selection

I guess you can see the picture ;-) Talking about the picture, here is one to show you what I am talking about.

The double list widget

When you start to look around for a better replacement, you obviously find the plugin sfFormExtraPlugin which includes the widget called sfWidgetFormSelectDoubleList and you get something like this:

This is already a big improvement but still has some huge limitation when you are dealing with big lists (I am thinking here stuff like list of students in a University). You do not want to have a select box with thousands of options: 1) it is going to slow down your page and 2) it is not easy to search inside.

A double list enhancement

The first "natural" step could be to improve an existing widget. In our case, the best is so far the double list widget mentioned earlier. An easy way to improve it could be to add a search box on top of the unassociated select in order to easily narrow down the list and find what we are looking for. We could even imagine a similar search box on top of the associated list to add a little bit more flexibility too. Anyway, here is a mockup that explains all this:

This could be already a huge improvement, although I can see still some limitations, namely: 1) you still load a huge list of unassociated items, 2) when you start filtering with your search, you need to select the desired item and then click on the left arrow, this is a bit too much interaction.

Anyway such a widget would be a huge improvement already and could be used in many cases, but let's not stop here...

Thinking forward

We came up with a mockup that suits our needs. I acknowledge that it cannot fit all needs, it is good to list a short list of relations -usually list of student in a class is under 40 and add new students among a huge list.

You basically search in an autocomplete box (ajaxified) to find the item you want to add, click the right one in the list and it is added to your checked list below. The all list is saved once you save the current form object. If you uncheck one item it will be removed from the list (with some delay to allow you to change your mind in case of a mistake). Finally we could add a local undo/redo function in case of mistake.

In terms of interaction, you type in your searched word, select from the list (click or enter) and you are ready to add another. I believe it can help a lot end users.
This symfony widget does not exist as far as I know therefore next part of this tutorial will focus on the code to make it a reality.

Conclusion

The first thing I'd like to claim: "multiple select" is BAD! At least I could not find a user-case where it could be a good solution, therefore avoid it as much as you can.

a new version for sfAdminThemejRollerPlugin 0.2

Finally! We finally managed to catch up and prepare a new version of our Symfony plugin to generate nice admin generator based on jQuery UI Themes.

We tagged this version as 0.2.0 since we still consider this plugin as beta version (but usable on production) but the big news is the support of symfony 1.4! For this specific support I must thank you 2 people that submited 2 patches to make this compatibility possible. Thanks to ( jtexier , "merci Jerome ;-)") and dalexandre and of course I must named jeremyb for his dedication to the project, he did most of the latest development.

I also setup a live demo to allow you to play a little bit with the plugin. It is currently basic but we will add more stuff step by step.

What's new:

• Theme switcher integrated
• user interface fixes and enhancements
• Symfony 1.4 compatibility

You can have some more details on my previous post.

The plugin is available as usual on github, I published as well the packaged plugin on symfony website.

SfGuardUser sync with Google Apps

As I am working for a university, we have recently migrate all our staff to use Google Apps for Education and we plan to move all our students and alumni too later this year. Google Apps for Education is really a great tool (and free for us ;-), most of our users are really happy of the switch: it is reliable, fast and have so many collaborative features. Bye bye Outlook :-(, welcome to the Cloud :-)
The Education package is the same as the Business package, including API and a lot of good stuff. The online Google Apps tools to manage users and groups is easy and simple but quickly become too simple to use when you are dealing with thousands of users, you then need to develop your own tool to maximize your daily operation. The "official" PHP5 Google Apps library is GData from the Zend Framework, which make it a breeze to integrate it with Symfony.

a view of an admin generator with Google Apps account management (using sfAdminThemejRollerPlugin):

We are currently redeveloping our University Management System based on Symfony 1.2 and we plan to integrate most of our applications with symfony too, including a research project on a new social learning platform.

I developed a Symfony task (sf1.2+Doctrine) that take a csv file and create your local sfGuardUser as well as the corresponding Google apps accounts. The Google accounts creation is optional so the task can be used for a normal batch user creation with Symfony alone. A simple:

./symfony guard:batch-create-user your.csv --googleapps=yourdomain.com

...will do the trick for you! I am not sure whether publishing the source code might be interesting for others (the Google API is only available for Premier and Education customers)...let me know. We will eventually publish our University management system as an open source project later for sure.

Here is a list of issues and recommendations while dealing with Google Apps Provisioning API:

• Google Apps API error reporting: when you have an error while dealing with Google Apps API, errors messages are useless, you always get a 'server errors' message without more details, it is then very difficult to debug your code to find the issue.
• Don't forget some of the basic Google Apps rules for account creation such as a password with minimum 6 characters otherwise you get back one of those unmeaning-full error message.
• sfGuard is using sha1 as the default encryption method but it adds a grain of 'salt' on it therefore you cannot send to Google Apps the correct sha1 password. I had to overwrite the default setPassword() and checkPassword() to simply remove the use of the salt and make sure my password synchronization with Google Apps can work. I am not sure if it is a huge security concern though, any idea? Beside this you can also use md5 for your password but you will not be able to use another algorithm since Google Apps is only accepting sha1 and md5.

what's up on sfAdminThemejRollerPlugin?

I finally found some time to add new stuff on the unpronounceable plugin, and thanks to some new contributors more are to come (thanks JeremyB ;-). Here is the list of change:

• As I mentioned in my previous post, I add a new config field named 'show' for the admin generator. I guess the meaning of this new option is quite obvious, isn't it? It adds the possibility to simply view one object details without editing it. It currently shows the list of fields like it is for the 'edit' view, meaning with a tab interface.
  
  
• UI improvements (done by jeremyb): as you can see in the 2 screenshots below there are many improvements such as the new 'reset' button next to the filter button, the sort arrows, new css styles for object actions, new pager view and finally the batch action select box is now also supporting the theme roller

The top part looks like this:

• notice the new 'reset' button next to 'filters'


• the up and down arrow on the table column names


• the new css style for the object actions

...and the bottom part (new paginator and the select box is also styled according to your roller theme):

the edit view (buttons are accessible both on top and button of the form, tabs are now vertical):

The current code is available in github only. It is not yet ready for a new plugin release, we are still missing some key elements.

new version of sfAdminThemejRollerPlugin (v 0.1.4)

I have finally released a new version of this plugin with an unpronounceable name ;-) Here is the list of change:

• I add a new field named 'show' for the admin generator. I guess the meaning of this new option is quite obvious, isn't it? It add the possibility to simply view one object details without editing it. It currently shows the list of fields like it is for the 'edit' view, meaning with a tab interface.


• I also create a new class jRollerDoctrineGenerator to replace sfDoctrineGenerator if you want the default link (the one you create with the = sign before the field name in the generator.yml) to link to the 'show' view rather than by default going to the 'edit' view. So far this is the only added value of this class. Of course if you prefer to keep the link to the 'edit' view simply keep the default class sfDoctrineGenerator. To use it, you simply need to modify your generator.yml as followed:

generator:
  class:     jRollerDoctrineGenerator
  params:
    theme:   jroller
....

The source code is available on github for the moment, I want to finalize a few things before packaging a new plugin version. I also need to put the source code on svn.

Propel ORM is dying?

as we used to say in French: "Propel is dead, long live Doctrine!". Indeed it seems that Propel development as slow down to a point where the lead developer quit the project (see his post). Guys, there are no more excuse to not look at Doctrine for all your Symfony project (and more).

New version for the symfony jRoller admin theme

I have just released the version 0.1.3 of the symfony plugin named (take a deep breath ;-) "sfAdminThemejRollerPlugin". I fixed a few inconsistency and allow to change to your own theme without the need to install your own version of jQuery and jQuery UI. Within the plugin there is only one theme named "redmond" (what a bad idea to name a theme like this :-( but you can download one from jqueryui.com and install it on your own web directory (see README file).
I have also uploaded it to github.com and to the official symfony svn repository.
Taste it and enjoy it!

sfAdminThemejRollerPlugin: Symfony admin generator on steroid

It's been a while since I wanted to work on my own admin generator theme. In symfony 1.0 the default admin generator was not really nice, it became a little bit better in 1.2 but still not to my taste (purely subjective I must admit ;-). For example, one thing get me mad is the css for the table list and the filter, I believe that the table list should be maximized and filters available only when we need them.
I spent quite some time to modify some admin generator in some projects but you always end up doing the same thing over and over (see this screenshot of what I have done with the list view based on a standard admin generator)...it was time to make things more to my taste. I now have a pretty clear idea of things that I would like out of the box from my new admin generator. Let me be more specific with the complete list of things I would like:

1. Change css style to a more modern style (again purely subjective ;-) that can be changed easily. For this I am using jQuery UI with their new theme roller system. The plugin is packaged with only one theme but it is extremely (too?) easy to change to your favorite theme (long term will be to use the upcoming html5 grid).


2. The filter box is hidden by default with a button to show/hide a modal window with the list of available filters. This allows to have a larger space for the table list.


3. In the new/edit views fieldsets are replaced by tabs (again jQuery UI).


4. Add a show only view, it makes sense for some application to allow read-only access.


5. filters ala iTunes smart playlist, meaning you select fields you want to filter and add them (nice to have but not so important, although useful if you have a lot of filter options)


6. Print preview (configuration in generator.yml): simply a non paginated table easy to print with sortable column (javascript grid style)


7. live search (configuration in generator.yml): be able to quickly filter the table list while typing a few characters, ajax style. I did that before for one specific module, I now have to make it part of the admin generator.


8. export to CSV format: we could have a modal window to select the fields we would like to get and export them or simply a list defined in the generator.yml (which can include getters that are not database fields).


9. export to PDF: does it make sense at all, I am not yet sure!


10. Import CSV to database: to be able to simply upload a csv file of data to import through the admin generator.

The first 3 items are easy to do just by modifying existing admin generator templates and are indeed already packaged in the plugin. The rest need more work (still learning how admin generator are working deeply).

I am also trying as much as possible to keep all actions accessible without javascript and make sure that I do not break any default admin generator css classes and ids in case you need them.

"Stop talking and show us the code"...here you go! It is published as a symfony plugin at http://www.symfony-project.com/plugins/sfAdminThemejRollerPlugin, meanwhile here are some screenshots:

How to install?

./symfony plugin:install sfAdminThemejRollerPlugin --stability=beta

When I say beta, it is beta ;-) I am not fully satisfied with the code and still thinking about the best approach for some features (not mentioning it's my first play with admin generator). Feel free to give me feedback.
I will publish the code on github later to ease collaboration if any of you are interested. Anyway, it's php right? so you can play with the code right now!

How to use it?

./symfony --theme=jroller doctrine:generate-admin

or if you have an existing admin generator, simply modify your generator.yml file with

theme: jroller

instead of theme: admin.

Configuration

I currently package jQuery 1.3.2 and jQuery UI 1.7.2 to make sure it can work out-of-the-box, but it is also easy to specify in your app.yml file that you do not want to use those versions (or because you already include those libraries in your project) and include your own jquery and UI.
jQuery UI is using at the minimum: Tabs, Dialog (Drag and drop and resizable recommended).

# inside your app.yml file
  sf_admin_theme_jroller_plugin:
    web_dir:      /sfAdminThemejRollerPlugin # specify specific folder where to pick your jquery stuff, ui and themes.
    use_jquery:   true # default. use the packaged jquery/UI
    theme:        redmond # default. 
    css_reset:    true # default. reset default css (from Blueprint CSS)

What's next?

I will try to implement the rest in the coming months (I just found out that sfDoctrineAdminGeneratorWithShowPlugin start to implement some of my ideas), again feel free to contribute to speed up the process.

Bugs? no way!

• I still have a pixel problem with the caption and tfoot of my table, if anyone knows how to solve it?

You will probably find bugs, please report them.

Symfony batch and i18n + pdf generation

A few months ago, I wrote a post about i18n in batch and pdf generation. At that time I could not find a way to manage translation in symfony batch (for example when you need to send email to customers, etc...). In Symfony 1.2, batches are depreciated in favor of tasks but my project is still in symfony 1.0 (and I guess the same can be applicable in 1.2). Anyway, I finally found a way to do it:

sfLoader::loadHelpers('I18N');
$I18N = sfContext::getInstance()->getI18N();
$I18N->setMessageSourceDir(SF_ROOT_DIR.DIRECTORY_SEPARATOR.SF_APP.DIRECTORY_SEPARATOR.'i18n', $culture);
echo __('text to translate');

I am not sure if it is the best solution, but it definitively works.

On the pdf side, I was mentioning that in order to generate invoices in pdf, I created some pdf invoice templates with Open Office Draw. Following this step, I was able with Zend_Pdf to generate PDF files on the fly based on my templates. The big problem with this solution is that you have to positionned all your text at the pixel and cannot manage things like multiple lines text and so on. You end up writing a lot of stupid code and anything you want to modify you have to try again and make sure it works (and when you manage several languages it can even be more complex). Anyway I managed to do it like this for the billing system used in East-Timor but I always wanted another solution. I did find some promising project but nothing yet reach a usable state to my knowledge. However, I found to interesting projects:

• odtPHP: it is not as such a pdf generator but it looks fantastic to generate open documents easily with a templating system. It is great and I am sure I will use it soon. Once you have an open document, it is a piece of cake for anybody to save it as pdf...even my grandma could do this ;-)


• Podr: it's a kind of open office server to generate document on the server, including pdf. However it seems only working on windows and quite heavy installation, I am not sure if it is a good solution. Anyone using it?

odtPHP looks like a clean and nice php class and it should be easy to integrate in symfony. On the client side, if you do not have open office (or any odt compatible solution) you can simply install a plugin for Microsoft Office to support open document formats (it seems it is even native support in Office 2007 service pack 2): http://www.sun.com/software/star/odf_plugin/ or http://sourceforge.net/projects/odf-converter/.

My view on CMS - the backend, part 1

CMS is a complex topic, everybody is talking about this and everybody has a different approach. I don't want to give my own concept here (should I have one ;-) but simply have a practical approach based on a project we have to do for one of our customer (largest casino in the world...). We did the first version of its CMS based on a typical Symfony 1.0/Propel/MySQL5.

We are now in the process to re-write the all application to make it useful for further projects and customers and optimize things based on our small experience. We decided to make the "big sf jump": symfony 1.2/Doctrine! It is not an easy choice since we did not have yet any experience neither with symfony 1.2 nor doctrine but we believed it was worth to make the jump specially due to the new admin generator and the form framework, not mentioning all the goodies from doctrine itself.

When you build a CMS, it is quite common to think first about the backend that will allow your customer to manage its content in an more-or-less easy to use interface. This interface is a key for a CMS success (along with its flexibility and extensibility). On my course to draw the backend interface I've tried to pick the best patterns (at least to me) from SPIP (a french CMS), Joomla and Wordpress. Thanks to Balsamic, I quickly draw what would be our CMS backend, at least for page management (the central part of the cms). See below for mockups.

List of pages

The list of pages should use symfony admin generator to keep its flexibility, therefore there is a lot of work on css and javascript fronts. A few things will become ajax (I will publish later my 2 cents jquery plugin to ajaxified a symfony admin generator list): pagination and the new live search.

For the list of filters, I wanted to make them less obstrusive, with some ideas from Google Docs search options (show and hide them) as well as Apple iTunes smartlist configuration. The latest will be a little bit more tricky to do...probably not for version 1!

For the need of a CMS we add the following elements to the default admin generator:

• Live search will "ajaxly" filter the list according to a search on title and content of the page.
• New batch actions such as "copy" and "move" pages.
• A left sidebar to navigate in a tree way (based on nested set). This navigation will filter the table in the same way as live search (ajax as well).
  

As we use Doctrine, we will use of lot of out-of-the-box behaviors: Nested set, versioning, i18n and of course Timestampable.

Next step we will look at the page edition view...see you there!

Invoicr (former CS-4U), what's new?

Long time I have not talk about my project that I used to name "CS-4U". As you might know, this web-based billing system for utilities, developed with my favorite PHP5 framework (should I mentioned it..symfony 1.0 at that time), is actually in use for about 3 years already in East-Timor for the electricity company.
I recently went back there to make an upgrade and add a lot of new features, this is what I want to share in this post.
This application was my very first project with symfony and when I dived again in the code for all those new features I realized how much I learned in the past years... There are lot of things that I would do differently today (and I am not even considering here the use of symfony 1.2)!

Here is a quick list of what has been added in version 1.1:

• PDF support with Zend_Pdf. This allow to generate invoices and reports in PDF (mainly using templates). Zend_Pdf is stable and easy to integrate in symfony, the drawback of these kind of solution is their limitation and the cost of maintenance: you must draw everything per line and position each line on the page. I created some simple methods to right align or draw a simple table over several pages but it is really a pain! It seems some guys are working on improving Zend_Pdf (they actually released an alpha framework RE) to add to it some more advanced functions such as multiline content, table and so on. I hope it can come out soon...
• A new receipt system. Basically the new receipt system will split a payment into sub-receipts over several invoices if necessary and still keep track of the "physical" payment main receipt. With this we clearly identify where is the money, there are no more "floating" money in the contract balance. All supplementary payments are made against the oldest debt of the contract and the system cannot accept payment over the total balance.
• Some new modules: Fines & Products module to add one-shot cost in upcoming invoice for a customer. Some admin generated modules to easily modify some of the data (status, districts, categories...)
  
• Barcode support. You can now add a barcode in the invoice pdf file to ease payment and avoid human errors. This is not yet use in Timor because they do not have yet barcode reader but they seems very interested by this solution.
• Javascript switch: from Prototype to jQuery. To ease some development I simply switch to jQuery library instead of Prototype and we use a little bit more ajax call in the app when it make sens. I still want to add more in the future...probably when I start to migrate to sf1.2...not for tomorrow though...
• New UI design. Not big changes but still an improvement compare to before. It is still very "light web style" since I am not yet convince to go to a more rich UI. I am kind of fan of "Google style".
  
• More actions to give more flexibility to users to correct mistakes mainly: change tariffs, modify meter-contract relation information, re-calculate invoice...
• Bugs, bugs, bugs.
  

What will come next (no particular order):
Business side:

• new tariff system to allow more complex tariff calculation.
• Dashboard. At login, user will see its own dashboard according to its authorization level. It will show some KPI and latest info/actions.
  
• Reporting. Change the reporting into a more user-controlled solution where the user can define the report she/he wants and get it in pdf (no idea yet on this ;-) or OpenDocument format.
  

Technology side:

• use of admin generator instead of manual dev for most of "master data" modules with a lot of ajax (more on this later as I am working on a new admin generator theme for this).
• Messaging Service. To ease communication between users or from system to users with memo, task assignment or system message. This is a symfony plugin I want to do for quite sometime already (including an event/calendar system too).
  
• Reporting. I want to make the reporting much easier and flexible for user to get the report he wants (not only pre-coded report). I also want to switch from the existing flash graph library to canvas based charts and/or open flash chart.
• Migrate to sf1.2 and write unit and functional test.

Symfony batch in my i18n nightmares

I was working on a new version of the billing system for utilities (used by the national electricity company in East-Timor), adding a bunch of new features including PDF generation for invoice and some reports. I decided to use Zend_Pdf and I believe it is a good choice.

Generating PDF is not something really fun. I used a template pdf and feed it with all the customer information I want. For each single piece of data you want to add, you must provide the position in the page, the style (if you want to change it)...a real pain! I wish we can have a more powerful templating system. It seems some guys have some plans (and codes) but nothing yet release (see mailing list here)...hey! (almost) good news: while I was looking for the link I found that they did release their PDF library (not yet in Zend, nor compatible with Zend_Pdf) named "RE Framework". I need to have a look one of this day...

Anyway, I was thinking to generate all invoices (in DB + PDF version) through a batch in symfony. I started working on it when I faced a "simple" problem: I need to use i18n for each invoices (invoice in the language of the customer), but from a batch I am always getting an error message since it does not have a user and culture define. I tried to initalize a fake one...but no luck. I need to try harder... ;-)

Any ideas?

Symfony 1.2, now available in shops!

Symfony 1.2 is finally officially out! As usual, it comes with a lot of very good documentation available.
After almost 3 years using Symfony 1.0, I've started to read some documentation for the new version and what I can say is simple: WOW!
I've really enjoyed to work with SF in the past years but I am now sure I will enjoy even more working with the 1.2 release...provided that I take the time to learn all the new "powers". Because this is the drawback of this version: coming from 1.0 you have to learn a lot of new stuff to master symfony. But don't get me wrong, I don't complain, I perfectly understand the need to learn more and all the new powers you get in hands after...like when Peter Parker discovered his spider powers ;-)
Coming from Symfony 1.0, here is the list of what impress me the most so far:
- the form framework
- the new admin generator based on the form framework
- the new routing system...still need to read more about it.
- doctrine integration. Doctrine is not new as such but I used propel before and want now to switch. Doctrine is amazing (see my previous post) and now fully integrated in SF.

Enough for today! I go back to my reading:-)
Meanwhile, SF team is doing again it's famous advent calendar based on 1.2... go job guys!

-- Post From My iPhone

Starting with Doctrine ORM

Preparing myself to switch from Symfony 1.0 to Symfony 1.2 as my default framework, I started to read the Doctrine documentation to get familiar with it - I used Propel as default ORM in Symfony before. I bet I am not the only one in the community...

I have not yet reached the end but I cannot resist to share my feeling about this amazing piece of code. As a symfony developer I do not need to read all Doctrine documentation since a lot of things are automatically generated thanks to the tied integration between the 2 frameworks.

Since I was used to Propel behaviors and installed in most of my projects some pretty standard plugins to create behaviors such as: Slug, Versionning, Nested set, Tags... I came to realize that most of those plugins are actually part of Doctrine out-0f-the-box! Just an example from the documentation to create a simple wiki with i18n (internationalization), slug, search and versionning (it could be apply as well to a CMS or whatever you wants):

WikiTest:
actAs:
 I18n:
   fields: [title, content]
   actAs:
     Versionable:
       fields: [title, content]
     Searchable:
       fields: [title, content]
     Sluggable:
       fields: [title]
columns:
 title: string(255)
 content: string

Even thought I will propably prefered to use Lucene for the search part, all the rest is amazing: just a few lines in your schema.yml files and "hop!" you got everything ready (I don't even mentioned that you actually nest those behaviors here...). They even have a "Geographical" behavior that I definitively want to test later.
It is also more readable and make more sens - to me at least- to define behaviors in the schema.yml file rather than in each class of the model.

There is no doubt, Doctrine is a good move! And remember: I am just talking about behaviors here, there are much more things in the documentation...I still need to go to read them...