Index to London Gazette - New to me

[paulmcmillan]

All

I found out there was Index to London Gazette was published I assume annually which lists all names by Index so making searching much easier - I have seen a couple with all RAF Appointments in various years - has anyone come across these before and know where they are available?

Paul

[Ross_McNeill]

There was a version available as an API dataset - eg an xml format that you can access via a remote program and interrogate with reply.

An early version of my gazette finder used this API to give better results than Google.

As time went on Google got better until it was on par with the dataset - but both are limited by the quality of OCR.

Later versions of gazette finder used 6 month and 12 month seniority promotions for AP/O, P/O and F/O to narrow down issues, used fuzzy logic to look for adjacent names to the one being searched so that poor OCR on a name could be combatted.

This is the link to the Longitudinal Dataset page from LG
https://www.thegazette.co.uk/data/longitudinal-datasets

Other national archives also provide API to their catalogue so I also used TNA discovery API in gazette finder as well.
https://www.nationalarchives.gov.uk/...interface-api/

Ross

[Ross_McNeill]

One of the personal databases I transferred to Jagan as a donation for RAFCommands to use recently was my generated Gradation Index that was the final refinement wip for gazette finder.

This was an off line excel database of RAF Officers.

1900-01-00 is a blank entry in the column

Sample

number

surname

initials

F1

F2

F3

F4

F5

F6

F7

MRAF

ACM

AM

AVM

AC

GC

WC

SL

FL

FO

PO

APO

WO

branch

courses

postnomials

AAFdate

RAFOdate

RAFVRdate

SpecialRdate

AAFROdate

MRAF-1

ACM-1

AM-1

AVM-1

AC-1

GC-1

WC-1

SL-1

FL-1

FO-1

PO-1

APO-1

WO-1

AAFdate-1

RAFOdate-1

RAFVRdate-1

SpecialRdate-1

AAFROdate-1

22170	Pace	R G	Richard	Griffith						1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1926-03-13	1900-01-00	1900-01-00	General			1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00											########
39243	Pace	T G	Thomas	Gilbert						1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1937-08-31	1936-11-02	1900-01-00	General			1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00											########	########
31493	Pacello	E A	Edward	Arthur						1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1939-02-01	General			1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00													########
36597	Pack	A T	Arthur	Thomas						1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1934-09-01	General			1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00													########
13068	Pack	G T H	George	Thomas	Harvey					1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1938-07-01	1935-07-01	1923-06-30	1919-10-01	1900-01-00	1900-01-00	1900-01-00	General			1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00							########	########	########	########
37070	Packe	G	Gordon							1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1937-12-21	1935-01-16	1900-01-00	General			1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00											########	########
	Packer	J P S	John	Philip	Stanley					1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1937-01-27	1936-03-23	1900-01-00	General			1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00											########	########
	Packham	A J	Alfred	John						1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1918-04-01	1900-01-00	1900-01-00	1900-01-00	General			1900-01-00	1922-10-24	1900-01-00	1900-01-00	1900-01-00										########					########
41730	Packington	R E	Raymond	Edwin						1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1939-03-04	1900-01-00	General			1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00												########
	Packman	S A	Stanley	Arthur						1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1925-05-05	1900-01-00	1900-01-00	1900-01-00	General		Major RARO	1900-01-00	1925-05-05	1900-01-00	1900-01-00	1900-01-00										########					########
	Paddle	M	Maurice							1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1938-07-08	1900-01-00	General			1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00												########
28097	Paddon	B	Brian							1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1938-08-01	1934-12-28	1900-01-00	1900-01-00	1900-01-00	1900-01-00	General	N		1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00								########	########
40744	Paddon	J W	John	Wreford						1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1900-01-00	1939-03-07	1900-01-00	1900-01-00	Admin			1900-01-00	1900-01-00	1939-03-07	1900-01-00	1900-01-00											########					########

It is on his to do list to bring into a format for the forum but has advised that a subset is incorporated into the linked databases.

"One small thing I was able to accomplish is to make use of the Gradatation data excels you sent. It is now part of the system (mostly) and appears in the 'details' pages time to time.


eg. - Guy Gibson - http://www.rafcommands.com/database/...php?qnum=16510 or http://www.rafcommands.com/database/...IRE&qnum=72021



I had to 'unpivot' the data in the sheet and then use a standard table with a key to link to the name records.. works for now."



May be more of a thing you want to use than the APIs.

Ross

[paulmcmillan]

Ross

Thanks for that I have been playing about with the URL format and have not got any results

I don't suppose you have one that can be used thanks ?

"The most user-friendly interface for longitudinal data is The Gazette browser. You can access this using the following format for URLs:

https://www.thegazette.co.uk/{edition}/index/year/{year}/volume/{volume}/page/{index page}?view=linked-data

Where:

{edition}: the edition of The Gazette index, eg London, Edinburgh or Belfast

{year}: the year of the index, eg 1909

{volume}: the volume of the index, eg 4

{index page}: the page of the index volume, eg 012 (must be 3 digits)

"

So for example, the illustrated values

https://www.thegazette.co.uk/London/...ew=linked-data

Does not work for me

Thanks Paul

[Ross_McNeill]

Yup - just checked it works in the forum GlobalDataBase yellow search box top banner.

eg put in
Pack

comes back with

https://www.rafcommands.com/database...php?qname=Pack

Select 13068

https://www.rafcommands.com/database...ACK&qnum=13068

Works a treat Jagan!

Ross

[Ross_McNeill]

Never had much luck with the HTML interface with Gazette so ended up writing my own script to build the query and submit as a JSON.

From memory the problems I had with HTML interface was difference between ASCII and numeric numbers - the API was very specific in the format and the browser interface tried to be to clever when I input - second guessing what it thought I meant before sending.

That along with wanting to be very specific in required search parameters led to the bespoke format.

Gazette Finder was a answer of it's time - mostly redundant now with what is available on RAFCommands under Jagan.

eg

F1180 for 22/2/33 has an accident where Moth K1219 was hit by taxying Bulldog K1638

Pilot is listed as Proctor, GEF? (indistinct), F/O, date of wings 1/3/29

Now you could try to find an AFL for the general date and look him up in the index, or see if you could find him in the google search/api for the London Gazette.

easiest way now is to put Proctor into the forum data base search bar -
https://www.rafcommands.com/database...?qname=Proctor

There is a GEF

https://www.rafcommands.com/database...TOR&qnum=27013

Looks a match for the wings date - we see he was transferred to RAFO shortly after the Bulldog accident, continued promotion in 1935.

Also from Jagans database we see what appears a return to colours with promotion to temp S/Ldr in 1941 retaining previous personal number.

A post from the archive in 2005 - 18 years ago! fills in the gaps and shows just how long posters here have been sharing knowledge.
https://www.rafcommands.com/archive/06164.php

All much easier than delving into the APIs.

The Gradation Database was included in the latest resource transfer to the Forum so that it can be used to resolve and be linked to newly generated databases being created now.

Oh - as a side note most LG entries are duplicated between London, Edinburgh and Belfast Editions and, due to differences of original print, an entry that fails OCR on London may be found OCR correctly on Edinburgh or Belfast editions. A trick I use to solve some OCR failures on London edition.

Ross

[Ross_McNeill]

Ok Paul,

Do not think the Gazette index is the asset you think it is but here is how to make the Longitudinal Dataset work and some other related bits.

Had a dig about in my old notes from conversations with the very helpful people at Gazette and TNA - they were astounded that someone wanted to access and use the systems - seemed up until my enquiry it was just a programmers vanity project.

Remove the ?view=linked-data

https://www.thegazette.co.uk/London/...ume/4/page/012

This defaults to the pdf set rather than HTML

On the gazette github is the dataset
https://github.com/TheGazette/Datase...909_4_0012.xml

For other years 1940 to 1949
https://github.com/TheGazette/Dataset-Index-1940-1949

Picked at random - the github index was not complete when I did gazette finder and still is not.

https://www.thegazette.co.uk/London/...ume/4/page/133

As you can see the index is name and page number - you still need Edition to be able to go to a particular entry. (Page number resets to 1 on first Edition Jan of each year).

eg Stenning TA page 5244

I did a look up table to get Edition for page range each year

Edition 36267

plug all that into the url format to go to the page
https://www.thegazette.co.uk/London/.../5244/data.pdf

As I say google has got better over the years so instead of my Edition lookup

google
Stenning 5244

gives the same hit.

Ross

[Ross_McNeill]

This is about the only way I found to use the Index dataset in any useful format.

Proctor gazette 1929 in google does not easily find him - wanders off into pages of Proctor & Proctor

Index to Gazette 1929 gives as page 7308
https://www.thegazette.co.uk/London/...ume/4/page/038

now
gazette 1929 7308 in google gives top hit
https://www.thegazette.co.uk/London/...3551/page/7308

It has a place to mine the depths for the nuggets that we cannot find easily elsewhere but it's not really supported by Gazette - github pages were last updated 9 to 14 years ago - and as you found help pages not up to date or correct.

Ross

[paulmcmillan]

Ross

Thanks for the tips - I wondered if I could say get a pdf lookup (not OCR) Index(es) for all promotions in a year and I believe I could get them but i think the art is finding where they are in the index which by trial and error for volume 4 1938 starts

https://www.thegazette.co.uk/London/...ume/4/page/045

End

https://www.thegazette.co.uk/London/...ume/4/page/053

So now i can download all them

Paul

[Ross_McNeill]

Yup - Military promotions at the end - in strict order of line etc RN, Army, Indian Army, RAF, Civil.

There are other sections earlier in each index for Gallantry and Honours as well as in the general index.

eg for 1938 DFC and DFM awards
https://www.thegazette.co.uk/London/...ume/4/page/004

leads to

https://www.thegazette.co.uk/London/.../7354/data.pdf

and

https://www.thegazette.co.uk/London/.../8076/data.pdf

some entries for Air Ministry names can be backdated by several years
https://www.thegazette.co.uk/London/...ume/4/page/002

https://www.thegazette.co.uk/London/.../6341/data.pdf

As a condition of getting Gov grant monies all institutions needed to make big data Public datasets as part of OGL (Open Gov Licence) - but apart from some insolvency and bankrupt datasets no one really made use of the Gazette datasets - that's why they were never progressed from the last update 9 years ago.

To show how powerful Public datasets can be if used as relational and linked datasets I recently used Defra Lidar datasets to get DTM (digital terrain model) into a 3d display package. Added OGL Ordinance Survey Building Footprint and Building Height datasets, Open streetmap for roads and finally used SUMO traffic simulation to produce a digital twin of 4 square kM of surrounding Twickenham Stadium with AI traffic according to time of day. All done using free open source assets.

That's why I think Jagan's approach on here is the way to go - link entries to the dataset enquiry to TNA, LG, CWGC. NAA, LAC, Canada Gazette etc and leverage the off site storage and resources of collections in a structured way to identify and show corrections. That way the catalogue errors that we see in AIR81 no longer drive errors into RAFCommands - rather RAFCommands presents all the alternatives and suggest the most probable solution from an AI knowledge base we add for RAF.

You can even suggest partnering to the LG to correct the OCR dataset when error found. The XML in the github is the tagged info for the page that the search engine harvest. You can strip it to give a machine generated list of the names on the page then using Mk1 eyeball correct this list and either do the XML update and upload to the github or request LG does the change and update the pdf.

Ross