The dual aspect of Drupal forms and what this means for your AHAH callback

Over the last week or two I've spent a lot of time on an aspect of my Quick Tabs module that I am certain none of its users will care a hoot about. It wasn't a case of adding a new feature or fixing a bug or even improving usability, but a question of, to put it succinctly, cutting down on its evilness. The admin form for creating and editing Quick Tabs blocks (where you choose either a block or a view for each tab) had a serious amount of ahah functionality: click a button to instantly add a new tab, click a button to instantly remove one of your tabs, select a view for your tab and have the view display drop-down be instantly populated with the correct options for that view. It was pretty user-friendly; there was just one problem: it flew in the face of Form API best practices.

When it comes to AHAH forms, there's the JavaScript side of things - where a behaviour is attached to, say, the onclick event of a button and new content is retrieved and inserted into the DOM - and then there's the PHP side of things, the AHAH callback where you rebuild your form with new or altered elements. Now, my understanding of FAPI voo-doo had been poor to begin with - I had competely copied poll.module for this side of things - and so when chx told me there was bad stuff going on in my AHAH callback and tried to point me in the direction of FAPI righteousness, I was pretty lost. Thankfully, he took the time to go through it with me, discovering that poll module was flawed in the same way, and though I didn't attain enlightenent straight away, the murk did eventually clear, I cleaned up my AHAH callback, and I think I have reached a stage where I can explain the crux of the problem to others.

You see, you need to think of your module's form (or any Drupal form) as being essentially dual in nature: there's the "material" form - what you actually see rendered on the page - and then there's the "spiritual" form, if you will, the form that exists out there in the ether (aka the form cache :-P). The material form and the spiritual form need to be representative of each other at all times. In fact, the material form should always be a "physical" manifestation of the spiritual form. If either gets altered without the other being altered accordingly, bad things will happen - at best, your form won't work properly, at worst, you will open the gates to FAPI Hell. For example, if you successfully add new elements to the spiritual form, so that the version of the form held in the cache now contains 4 poll choice textfield elements, but the material form, the form rendered on the page, still only has three poll choice textfield elements, then when you hit submit it will give you a validation error to the effect that you have left the fourth field blank, assuming such validation has been applied. On the other hand, say for example you have a dependent dropdown, where you populate the options in one dropdown based on a selection in another, if those new options don't exist in the spiritual form, you will get an error to the effect that "an illegal choice has been detected" when you submit. There are good and bad ways around this latter problem.

One way, the way many modules up to now, including my own, were doing it was to retrieve the form from the cache, tack on the new element, re-save it to the cache, rebuild the form and render the altered portion. This is illustrated below:

<?php
function myform_ahah() {
 
$delta = count($_POST['addable_elements']);

 
// Build our new form element.
 
$form_element = _create_new_element($delta);

 
// Build the new form.
 
$form_state = array('submitted' => FALSE);
 
$form_build_id = $_POST['form_build_id'];
 
// Add the new element to the stored form. Without adding the element to the
  // form, Drupal is not aware of this new element's existence and will not
  // process it. We retreive the cached form, add the element, and resave.
 
if (!$form = form_get_cache($form_build_id, $form_state)) {
    exit();
  }
 
$form['my_ahah_wrapper']['addable_elements'][$delta] = $form_element;
 
form_set_cache($form_build_id, $form, $form_state);
 
$form += array(
   
'#post' => $_POST,
   
'#programmed' => FALSE,
  );

 
// Rebuild the form.
 
$form = form_builder('mymodule_form', $form, $form_state);

 
// Render the new output.
 
$form_portion = $form['my_ahah_wrapper']['addable_elements'];
  unset(
$form_portion['#prefix'], $form_portion['#suffix']); // Prevent duplicate wrappers.
 
$form_portion[$delta]['#attributes']['class'] = empty($form_portion[$delta]['#attributes']['class']) ? 'ahah-new-content' : $form_portion[$delta]['#attributes']['class'] .' ahah-new-content';
 
$output = theme('status_messages') . drupal_render($form_portion);

 
drupal_json(array('status' => TRUE, 'data' => $output));
}
?>

The main problem here is that in between the form being re-saved to the cache and it being rebuilt, it is further altered by the lines

  $form += array(
    '#post' => $_POST,
    '#programmed' => FALSE,
  );

$_POST should not be used at all in the rebuilding of the form. Everything should come from $form_state, so that the rendered output is still a "physical manifestation", as I put it earlier, of the form in the cache.

In order to achieve this, a couple of things need to be done differently, the ahah callback needs to be altered - but this actually involves mostly removing code; and the function that generates the form needs to be structured so as to react to the contents of $form_state (when building the form it first checks $form_state for previously submitted information, then checks if it is receiving information from the database, then finally if it is not receiving information from anywhere, it renders a brand new clean form). The ahah callback then looks like this:

<?php
function myform_ahah() {
 
$form_state = array('storage' => NULL, 'submitted' => FALSE);
 
$form_build_id = $_POST['form_build_id'];
 
$form = form_get_cache($form_build_id, $form_state);
 
$args = $form['#parameters'];
 
$form_id = array_shift($args);
 
$form['#post'] = $_POST;
 
$form['#redirect'] = FALSE;
 
$form['#programmed'] = FALSE;
 
$form_state['post'] = $_POST;
 
drupal_process_form($form_id, $form, $form_state);
 
$form = drupal_rebuild_form($form_id, $form_state, $args, $form_build_id);
 
$form_portion = $form['my_ahah_wrapper']['addable_elements'];
  unset(
$form_portion['#prefix'], $form_portion['#suffix']); // Prevent duplicate wrappers.
 
$output = theme('status_messages') . drupal_render($form_portion);
 
drupal_json(array('status' => TRUE, 'data' => $output));
}
?>

The form is retrieved from the cache, processed, and rebuilt. During rebuilding, $_POST gets destroyed and the form is re-saved to the cache. You can see that there is no room for alterations to the rendered form that aren't mirrored in the cached form - so, then, where do changes to the form take place? Well, changes happen not in the ahah callback but in the submit handler for the element that triggered the callback. Your form is going to get completely rebuilt, so in the submit handler, which gets called during drupal_process_form, you can specify exactly how you want it rebuilt. Here's my submit handler for the "Add tab" button in quicktabs:

<?php
function qt_more_tabs_submit($form, &$form_state) {
  unset(
$form_state['submit_handlers']);
 
form_execute_handlers('submit', $form, $form_state);
 
$quicktabs = $form_state['values'];
 
$form_state['quicktabs'] = $quicktabs;
 
$form_state['rebuild'] = TRUE;
  if (
$form_state['values']['tabs_more']) {
   
$form_state['qt_count'] = count($form_state['values']['tabs']) + 1;
  }
  return
$quicktabs;
}
?>

There are two important steps here: we're passing all the form values (what has been entered on the form) to $form_state['quicktabs'] so that when the form-generating function is called, it will check here first to see if there are already values available for the elements it's building; we are also incrementing the number of tabs by 1 and passing this number in $form_state['qt_count'], which is where we tell our form function to check first for the number of tabs to build. [Note: when I'm talking about tabs here I'm referring to sets of elements, where each set corresponds to a tab of content in a final Quick Tabs block - not to be confused with the myriad other meanings of the word in Drupal]. With those in place we know that the entire form can be rebuilt from scratch, will be rendered faithfully on the page, and will contain everything we need it to contain, new elements and all.

Here is the submit handler for the views dropdown on the quicktabs admin form:

<?php
function qt_get_displays_submit($form, &$form_state) {
  unset(
$form_state['submit_handlers']);
 
form_execute_handlers('submit', $form, $form_state);
 
$quicktabs = $form_state['values'];
 
$form_state['quicktabs'] = $quicktabs;
 
$form_state['rebuild'] = TRUE;
  return
$quicktabs;
}
?>

Notice that it doesn't seem to make any change to the form at all - the form is simply going to be rebuilt but with a different selected value for this dropdown (coming from $form_state), and that will change the options in the display dropdown. The views dropdown uses the very same ahah callback function as the "Add tab" button, as does the "Remove tab" button - they only differ in their submit handlers. The code is therefore cleaner, easier to maintain, and most important of all - more secure.

So, despite its dual nature, with its "material" side and its "spiritual" side, the Drupal form can attain perfect oneness when this approach is taken :-)

To see the full Quick Tabs code, visit http://drupal.org/project/quicktabs and download the latest dev snapshot of the 6.x-1.x branch (2.x branch to be updated shortly). To see the form in action, click here. You will need to change the tab type to "View" in order to see the views display dropdown working.

Further Reading:
Doing AHAH Correctly in Drupal 6 and beyond

mistresskim (not verified) on November 29th 2008

Great post. If only all FAPI documentation were equally thorough! :) The code above works fine for me on my first iteration of AHAH but when clicking a button that was added via the first AHAH callback, I end up with an empty $_POST, and hence no form_build_id to get the form from the cache. Have you experienced this problem at all? Any thoughts/advice appreciated.

Philip Blaauw (not verified) on November 18th 2008

Hi Katherine,

Thanks for the nice writeup of AHAH functionality in Forms.

Developing the Dynamic display block module I also used some AHAH Functionality in the admin pages for different form content depending on selected options. We are now developing the next release of the module and have a lot more dependencies on selected options and I didn't like the way using the AHAH functionality used in modules like POLL and some other modules and what is used in views is at the moment to difficult for me. Also looked at the AHAH Helper module which looks like a great help in developing AHAH functionality although I have problems with setting different default values with this module, and it is not much maintained. I think your writeup will help a lot.

Also liked your writeups about JQuery functionality in Drupal. Used it also in the Dynamic display block module.

Thanks again, great help

Philip

Wim Leers (not verified) on November 19th 2008

It's being maintained, but I haven't had the time to handle support requests. Sorry for that.

I am responding to bug reports, of which there has only been one since the initial release. So I think "not much maintained" is a slight overstatement :)

I'm replying here because I wanted to put the right nuance in place; I'd like to see more people using this module so I can get more feedback, because it is the direction core is headed.

P.S.: I'll try to reply to your support request this weekend.

Philip Blaauw (not verified) on November 20th 2008

Hi Wim,

Thanks for your replies.

To make other people understand how to use your module, will have a positive effect on how many users will actually use your module and will make your module better.

I think your module is a great help in making AHAH Forms.

Like this writeup from Katherine to make developers understand how to make AHAH forms is very helpfull. (It's difficult to find the right way in handling AHAH forms, if you look at other modules how they use AHAH, it can easily put you in the wrong direction. The same you can say for the use of jQuery with Drupal)

AHAH forms are very user friendly for the user of the form but till now very unfriendly for the developer of the forms.

Thanks to you it becomes easier for the developer also.

Thanks again,

Philip

Anonymous (not verified) on November 18th 2008

Isn't it utterly insane that every single AHAH callback has to do this whole dance itself? Repeated code = bloated code. Given the simplicity of basic Form API use elsewhere (i.e. drupal_get_form + two callbacks), the effort required to add this supposedly 'easy' AHAH stuff is disproportionately large.

Wim Leers (not verified) on November 18th 2008

The AHAH helper module (written by myself, sponsored by Mollom) brings disaster relief here, it'll prevent you from entering FAPI Hell.

For example, all of this:

or example, if you successfully add new elements to the spiritual form, so that the version of the form held in the cache now contains 4 poll choice textfield elements, but the material form, the form rendered on the page, still only has three poll choice textfield elements, then when you hit submit it will give you a validation error to the effect that you have left the fourth field blank, assuming such validation has been applied. On the other hand, say for example you have a dependent dropdown, where you populate the options in one dropdown based on a selection in another, if those new options don't exist in the spiritual form, you will get an error to the effect that "an illegal choice has been detected" when you submit. There are good and bad ways around this latter problem.or example, if you successfully add new elements to the spiritual form, so that the version of the form held in the cache now contains 4 poll choice textfield elements, but the material form, the form rendered on the page, still only has three poll choice textfield elements, then when you hit submit it will give you a validation error to the effect that you have left the fourth field blank, assuming such validation has been applied. On the other hand, say for example you have a dependent dropdown, where you populate the options in one dropdown based on a selection in another, if those new options don't exist in the spiritual form, you will get an error to the effect that "an illegal choice has been detected" when you submit. There are good and bad ways around this latter problem.

Well, this would simply have been impossible if you'd use that module :) Read my blog post about it for details, and to find out why dynamic (AHAH-powered) forms in Drupal 6 are flawed.

katherine on November 19th 2008

Hi Wim,
I am of course familiar with your AHAH helper module - I have actually re-used some of the code from it in Quick Tabs (with proper credit to you in the README file) - specifically the override of the ahah success function, because I need ahah behaviours attached to the new elements. However, having a dependency on your module just wasn't an option because it's just for the admin interface - AHAH helper should be in core ;-) Also, the technique outlined in my post above is different from yours as it calls drupal_process_form() and drupal_rebuild_form(), which is different from your approach which does:

  // Now, we cache the form structure so it can be retrieved later for
  // validation. If $form_state['storage'] is populated, we'll also cache
  // it so that it can be used to resume complex multi-step processes.
  form_set_cache($form_build_id, $form, $form_state);

  // Set POST data.
  $form['#post'] = $_POST;
  $form['#programmed'] = TRUE;

  // Build the form, so we can render it, or continue processing it (call
  // validate and/or submit handlers)
  $form = form_builder($form_id, $form, &$form_state);

Anyway, I know there has been some discussion about getting an ahah callback utility function into core, AHAH Helper module being the most obvious starting point, so I look forward to following its progress. Do you think it'll make it into D7?

Katherine

Wim Leers (not verified) on November 21st 2008

My technique is not as much a technique as it is a hack. It's possible that there are cleaner ways to do this, but if there are, they're way too hard to find, and then FAPI itself is too blame. I still think $form['#cache'] (i.e. caching forms in the database) is useless and that FAPI itself is overly complex.

This is likely to make it into D7 core, since it seems the code freeze won't be even close in February 2009, which is when I'll have time again.

The real issue to tackle here is to make FAPI elegant (by which I mean the code should be much easier to understand). Would you be interested to help out in that area? :)