ย Back to homepage

Nested Promises and Promises in Loop

Published on 26/09/2020
Control the flow to run in a linear sequence.

Find the gist ๐Ÿ‘‰ here

import slugify from "slugify";
import axios from 'axios';
import db from "../components/db";

const localEventSources = [
    {
        url: "https://www.ilovetakapuna.co.nz/index.php?option=com_calendar&theAction=exploreNorthShoreEventsFeed",
        source: "takapuna",
        domain: "https://www.ilovetakapuna.co.nz/media/event/",
    },
    {
        url: "https://www.milfordshops.co.nz/index.php?option=com_calendar&theAction=exploreNorthShoreEventsFeed",
        source: "milford",
    }
];

async function crawler(event: any) {
    return new Promise(((resolve, reject) => {
        axios(event.url).then((response: any) => {
            let promises: Promise<any>[] = [];
            // eslint-disable-next-line promise/always-return
            for (const item of response.data) {
                const promiseReference = new Promise((resolve, reject) => {
                    db.collection("local-events")
                        .doc(slugify(item.event_title, {lower: true}))
                        .set({
                            title: item.event_title,
                            region: event.source,
                            startDate: item.event_start_date,
                            endDate: item.event_end_date,
                            imageUrl: event.domain + item.event_image,
                            simpleDesc: item.simple_event_desc ?? '',
                            desc: item.event_desc,
                        }, {merge: true})
                        .then((writeResult) => {
                            console.log('Crawling ' + item.event_title);
                            resolve(writeResult);
                            return writeResult;
                        })
                        .catch(e => {
                            reject(e);
                        });
                });
                promises.push(promiseReference);
            }
            console.log('Collected promises from one source');
            return Promise.all(promises).then(() => {
                console.log('Finishing promises from one source');
                resolve();
                return;
            });
        }).catch((e) => {
            reject(e);
        });
    }));
}

async function cleanUpLocalEvents() {
    const date = new Date();
    const todayISODate = new Intl.DateTimeFormat('fr-ca').format(date);
    const querySnapshot = await db.collection('local-events')
        .where('endDate', '>', todayISODate).get();
    querySnapshot.forEach(doc => {
        console.log('Deleting: ' + doc.ref.id);
        doc.ref.delete();
    })
}

export async function crawlerLocalEvents() {
    let promises: Promise<any>[] = [];
    localEventSources.forEach((source) => {
        console.log('Parent: collecting promises from a source');
        promises.push(crawler(source));
    });
    console.log('Parent: Collected promises from all sources'); // <<<<<<
    // console.log(promises);
    Promise.all(promises)
        .then(() => {
            console.log('Parent: all promises done');
            console.log('Cleaning up old entries');
            return cleanUpLocalEvents();
        }).catch();
}

/**
Console Logs - Almost correct sequence
i  functions: Beginning execution of "update_local_events"
>  Parent: collecting promises from a source
>  Parent: collecting promises from a source
>  Parent: collecting promises from a source
>  Parent: Collected promises from all sources
i  functions: Finished "update_local_events" in ~1s <<< ๐Ÿ“ well... this line maybe should be at the very end
>  Collected promises from one source
>  Collected promises from one source
>  Collected promises from one source
>  Crawling Bruce Mason Centre - Jazz Gala Concert 2020
>  Crawling Eric Verdonk Memorial Regatta - Lake Pupuke
>  Crawling Bruce Mason Centre - Te Moana Glow Show!
>  Crawling PICK 'A' BOX retail customer promotion
>  Crawling The Pumphouse: Tricks 'n' Treats Magic Show
>  Crawling Devonport : Architecture of Land and Sea
>  Crawling Bruce Mason Centre - The Nutcracker and Don Quixote
>  Crawling Learn to Dance - Takapuna with Modern Jive
>  Finishing promises from one source
>  Finishing promises from one source
>  Finishing promises from one source
>  Parent: all promises done
>  Cleaning up old entries   <<< ๐Ÿ“This one was almost at the top before I wrap actions in new Promise
**/

The basic idea is

  1. Wrap the actions should be queued in a Promise
  2. Collect their references to an array
  3. Use Promise.all( ) to monitor their status
  4. Run clean-up or something else when Promise.all() resolves

You can also capture that Promise.all() in step 3 to an array and monitor that with another Promise.all().

Be the first
to get my latest blog
Comments