Projects are (randomy) missing from update status API response

Created on 11 July 2023, 12 months ago
Updated 14 July 2023, 12 months ago

As you may already aware, I am running daily Drupal project scanners on Github Actions: https://github.com/mxr576/ddqg. This project heavily relies on the https://updates.drupal.org API.

Problem/Motivation

I have spotted yesterday when I was actively working on DDQG that after each commits some projects disappeared and re-appeared randomly on some outputs generated by the tool. This did not make sense because my commits were infra related only.
I did some digging today and it seems to me that those projects (see a couple of names below) are not always returned by https://updates.drupal.org/release-history/project-list/all. I have also attached a compressed output of the API, search for the project ids of these projects (they should be missing):

* https://www.drupal.org/project/chatbase โ†’
* https://www.drupal.org/project/sector_webform โ†’
* https://www.drupal.org/project/ajax_placeholder โ†’

Can you please investigate and provide some explanations about what could be the reason behind this behavior?

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

๐Ÿ› Bug report
Status

Active

Version

2.0

Component

Releases

Created by

๐Ÿ‡ญ๐Ÿ‡บHungary mxr576 Hungary

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @mxr576
  • ๐Ÿ‡ญ๐Ÿ‡บHungary mxr576 Hungary
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States drumm NY, US

    That is generated by https://git.drupalcode.org/project/project/-/blob/2b1c835a538ffcf2b50cbe...

    Iโ€™m not spotting anything that looks wrong right away. There are some INNER JOINs that could miss and remove rows. But that shouldnโ€™t be happening for your examples.

    This all projects list is not actively maintained. We may want to sunset it in favor of newer APIs. Where did you find documentation about it?

  • ๐Ÿ‡ญ๐Ÿ‡บHungary mxr576 Hungary

    Where did you find documentation about it?

    https://www.drupal.org/drupalorg/docs/apis/update-status-xml โ†’

    We may want to sunset it in favor of newer APIs.

    Could be a good idea, but that would also require refactoring in the update.module.
    Plus even if the returned XML by that API is huge, fetching that is still more performant than paginating over multiple pages of Drupal API. My project's build time has increased from 2-3 minutes from 12+ minutes.
    Before: https://github.com/mxr576/ddqg/actions/runs/5533272257
    After: https://github.com/mxr576/ddqg/actions/runs/5533395391

    I also have a more detailed list about missing projects in Update status API response:

    Array
    (
        [423] => adisandbox12
        [780] => ajax_placeholder
        [929] => advert
        [3195] => browser_force_logout
        [3280] => brevo
        [3697] => campay
        [3770] => captchasandbox
        [3958] => chatbase
        [4230] => ckeditor5_fullscreen
        [4515] => cloudflare_sitemap
        [5563] => commerce_graphql
        [5732] => commerce_funds_stripe
        [6069] => commerce_sandbox
        [6298] => commerce_trustpay
        [6341] => commerce_stripe_checkout
        [6524] => config_duplicator
        [6910] => content_dump
        [6916] => content_entity_sync
        [7159] => cookie_consent_notice
        [7193] => copyright_notice
        [7199] => copy_formatter
        [7213] => convert_currencies
        [7411] => culturefeed
        [7412] => culturefeed_agenda
        [7413] => culturefeed_api
        [7414] => culturefeed_content
        [7416] => culturefeed_search_api
        [7417] => culturefeed_user
        [7444] => currency_layer_integration
        [8046] => dark_responsive
        [8080] => dcge
        [8122] => dataflow
        [8184] => dblogall
        [8275] => daterangepickerwidget
        [8792] => discount_estimator
        [8839] => document_ocr_ai21
        [8958] => domain_perm
        [9253] => drupal_sheetjs
        [9281] => drupdate
        [9374] => drupal_js_path
        [9861] => easy_install_d10
        [9979] => eca_commerce
        [9984] => eca_metatag
        [10629] => entity_debug
        [10719] => entity_expand
        [10731] => entity_field_capitalization
        [11292] => extension_reference_field
        [11992] => festeva
        [12014] => fformatter
        [13437] => g2sandbox
        [13625] => generic_layout
        [13690] => ga_dashboard
        [14356] => graphql_book
        [16613] => jsonrpc_intl
        [16961] => language_switcher_browser
        [17019] => lb_copy_section
        [17069] => layout_builder_instant_preview
        [17126] => layout_block
        [17496] => library_generator
        [19136] => migrate_sandbox
        [19207] => meta_conversions_api
        [19505] => mobile_money
        [19756] => modal_menu_block
        [20067] => multi_region
        [20186] => neutrino_api
        [20219] => news_ticker
        [20232] => nextcloud_dam
        [20927] => ntc
        [20949] => number_in_word
        [21090] => octavia_camouflage
        [21145] => oa_sandbox
        [21622] => openid_connect_daccount
        [21978] => otp_service
        [22134] => pagination_manager
        [22324] => partytown
        [22851] => pdb_twig
        [23443] => power_portfolio
        [23795] => project_wiki
        [25656] => rules_flow
        [25821] => sandbox001
        [25822] => sandbox_ftw_promotion
        [25823] => sandbox_theme
        [25999] => sdc_display
        [26024] => search_api_external_index
        [26224] => schemadotorg_starterkit_podcast
        [26234] => schema_diff
        [26253] => sector_social_follow
        [26254] => sector_social_share
        [26258] => sector_webform
        [26664] => sfds
        [27289] => site
        [27290] => site24x7
        [27417] => site_manager
        [27788] => sober
        [28138] => ssom
        [28326] => state_token
        [28929] => symfony_mailer_lite
        [29154] => taxonomy_sitemap_xml
        [29633] => theme_system_sandbox
        [29821] => time_ticker
        [30033] => tmgmt_openai
        [30153] => token_aliases
        [30203] => top_description
        [32709] => views_filter_param
        [33058] => views_samefield_contextual_default
        [34537] => wxt_chart_stability
        [34787] => yaml_sandbox
        [34993] => y_camp
        [34995] => y_facility
    )
    
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States drumm NY, US

    I donโ€™t see /project-list/all mentioned at https://www.drupal.org/drupalorg/docs/apis/update-status-xml โ†’ and Drupal Core does not use the all projects XML. The individual project XML files will be maintained indefinitely.

  • ๐Ÿ‡ญ๐Ÿ‡บHungary mxr576 Hungary

    Well, then good question... :) I do not remember. Probably it was used/used to be used one of the tools that I used as inspiration.

Production build 0.69.0 2024