Iterators for SPIP - the (DATA) loop

SPIP 3.0 brought a brand new kind of loops, based on PHP’s iterators.

Not only (DATA) loops can execute SQL requests [1], but they can run on every kind of data lists.

For example:

  • a data array produced by any function (including PHP iterators)
  • a local file’s content, format XML, CSV, JSON, YAML, ...and so on.
  • a files list in a server’s directory
  • a web service request
  • a SQL request (e.g. as calculated by SPIP)
  • and so on. (LDAP, ...).

(DATA) loop’s syntax and formats

That loop can iterate on any data array. It’s basic syntax is as follows:

<BOUCLE_example(DATA){source format, data}>
	#TAGS
</BOUCLE_example>

The {source format, data} criterion defines the data on which the loop should iterate.

A data source definition needs two elements:

— The data part: this element can be of various natures:
-  a data array, for example: #ENV*
-  path to a file on the hard drive, example: sources/definitions.csv
-  a file or a web service’s URL, for example: http://per.sonn.es/bases/phobia.fr.yaml
-  or any other string the format will be able to convert to a data array, ex: "select * from flickr.photos.search where text='spip'"

— The format part is to be chosen from the following list:
-  table (a.k.a. array or tableau), for a previously declared array
-  csv, json, yaml for any file using one of those formats
-  file to loop on some file’s lines
-  glob or pregfiles to loop on files in a given directory (and more...)
-  rss (a.k.a atom) to read any news feed
-  plugins to list all enabled plugins on the website
-  yql to query on Yahoo Query Language web service
-  sql to send a raw request to the SQL server (you should use {source sql, connector:query} to query an external database)
-  ics to loop on calendars (needs icalendar plugin: read Plugin iCalendar for more)
-  and so on.

All those formats are already available and it’s very easy to add new ones, creating a simple function inc_FORMAT_to_array($u). As an example, here is the function that converts a JSON file to an array:

function inc_json_to_array_dist($u) {
  if (is_array($json = json_decode($u))
  OR is_object($json))
    return (array) $json;
}

We can declare a new (DATA) loop in a inc/my_source_to_array.php file

function inc_my_source_to_array_dist($data,$param1='',$param2='') {
     // $data contains (local | distant) file's content or, variable's value transmitted in criterion.
    // $param1, $param2... are optional parameters
...
}

and then display it:

<BOUCLE_example(DATA){source my_source,test}>
	#VALEUR
</BOUCLE_example>

Good to know: A (DATA) loop’s result is cached. We can disable this behaviour with {datacache 0} criterion.

Filtering, ordering, paginating, merging

Filtering. Like any other SQL loops, (DATA) loops can be filtered with criteria like {valeur=x}: available operators are =, >, <, >=, <=, == (rational expression) and LIKE

However, this filtering is not applied beforehand — during the request, like we do in SQL — but afterwards, on the table of data initially returned.

Ordering. All kind of {par xx} orderings are also possible, with their variant {!par xx} to backward orders.

Paginating. Pagination works normally, so does the offset/limit {a,b} criterion.

Merging. The {fusion /x/y} criterion also works. For example, on a CSV file containing addresses, if the email field is numbered 3, we’ll then be able to keep only one record per email address with the following loop:

 <BOUCLE_csv(DATA){source csv, addresses.csv}{fusion /3}{par /0}{'<br/>'}> 
 	#VALEUR{0}: #VALEUR{3} 
 </BOUCLE_csv> 

Merging is done after ordering and, only retains the first element it met. That way, if an array is ordered by {!par date} then merged with the email field, the entry that will be retained for each email address, will be the latest one.

{liste ...} criterion

To simplify the writing of data arrays, when it comes to a simple list of items we want to manually provide, the (DATA) loop also accepts {liste ...} criterion, which let you build an array, separating each data with a comma ",".

Loop:

<BOUCLE_i(DATA){liste 3,4,5}{"<br>"}>
	<BOUCLE_j(DATA){liste 6,7,8}{" "}>
		[(#VALEUR|mult{#_i:VALEUR})]
	</BOUCLE_j>
</BOUCLE_i>

returns:

18 21 24
24 28 32
30 35 40

Note: We used #_i:VALEUR in j loop, as a reference for the value calculated by i loop.

See also Two simple iterators : lists and enumerations

{enum ...} criteria

<BOUCLE_enumerate(DATA){enum 2,10,2}>
      #VALEUR
</BOUCLE_enumerate>

returns:

2 4 6 8 10

<BOUCLE_enum(DATA){enum g,m}{", "}>
	#VALEUR
</BOUCLE_enum>

returns:

g, h, i, j, k, l, m

val1 and val2 are two numeric values or, 2 characters. SPIP finding out which one, of the 2 values, is the smallest, this loop is going to enumerate values between val1 and val2. In the first variant, no interval is set: it’s worth 1 by default.

See also Two simple iterators : lists and enumerations

Protecting iterated data

(DATA) loop automatically protects every tag inside it, because the data we carry on often comes from unknown, potentially harmful sources. This also affects forms tags (#FORMULAIRE_xx) that you’ll have to call with a "*", as explained in the following article on #TAG* and #TAG**

<BOUCLE_foreach_protect(DATA){liste a,b,c,d,e}>
	#FORMULAIRE_XXX*{#VALEUR}
</BOUCLE_foreach_protect>

Note that, if you use the "Bonux" plugin, it’s going to take a simple loop (POUR), which does not have that behaviour.

<BOUCLE_Foreach_non_protect(POUR){liste a,b,c,d,e}>
	#FORMULAIRE_XXX{#VALEUR}
</BOUCLE_Foreach_non_protect>

 

Read more: have fun with some BOUCLE(DATA) loop examples !

You can also read the dedicated mini-website on SPIP and DATA loop uses
http://spip-loves-opendata.spip.net...

Footnotes

[1With no surprise, a classic SPIP’s iterator is named SQL: it executes the request, as SPIP calculated it and, knows how to browse results list to send them to the loop.

Author Fil, jack, Loiseau2nuit, Matthieu Marcillaud Published : Updated : 17/05/20

Translations : عربي, English, français, Nederlands