While working on the PGweb Testing Harness we noticed that although our testing suite works on checking the entire website for broken links, we are not able to check the real time data that is present on the main website since the testing harness uses a development environment with some predefined fixtures which does not aptly satisfy this particular test.
Therefore we have two patches to tackle the problem of Broken URLs at least in the dynamic content for the pgweb repository.
1. URL validation in URL specific fields and markup fields in the forms for adding Events, News, Organisation, Products and Professional Services.[form_validation_with_markdown_v03.patch]
2. A secondary service which would be run as a cronjob on the prod server to check tables that consist of specified URL fields and Markup content fields. (I am currently using the local values for database connection in the script; changes will be made according to the prod server if we decide to deploy it.)[url_scan_individual_tables.patch]
Patches have been attached in the above mentioned order. Would love to hear your thoughts on this. I think this can help us a lot in manually searching for broken URLs on the website (at least in the dynamically generated content). I am also working on refining the crawling service already implemented in the Testing Harness which would be able to check for broken URLs.