Universal Patcher-Beta issue with black box os

[CDNL]

I loaded osid 16263425 into Universal Patcher-Beta version 0.23.4.0 and Universal Patcher-Beta version 0.23.12.0 and got two different looking main spark advance tables. It seems that its the same for other black box osids that I tried. See Attachments.
Also, version 0.23.4.0 will not load osid 16265175, but version 0.23.12.0 loads it without problem.
Just putting this out there for information purposes.

[sparepartsblazer]

The rowstride and elementstride tables where renamed minorstride and majorstride, so it broke the xml. Try these.

[CDNL]

Once again sparepartsblazer, I'm very grateful for your help and the file you provided. And I want to thank you for the work you've done with black box pcms. I made the changes with the file you provided. Then loaded the black box osids that I have, checked the main spark tables and they all looked good. From someone still playing with the black boxes. Again, thanks for your help.

[CDNL]

It looks like the maximum knock retard table (rpm x map) has the same issue.

[sparepartsblazer]

It looks like the maximum knock retard table (rpm x map) has the same issue.

I will have to take some time a go though them and fix them.

[antus]

Sounds like a job for ai or an ai tool. Where are the XMLS, are they in the official universal patcher? If you are not setup for AI but you can list the XMLs that need a look I can feed them in to one and send it back to the universal patcher project.

[sparepartsblazer]

Sounds like a job for ai or an ai tool. Where are the XMLS, are they in the official universal patcher? If you are not setup for AI but you can list the XMLs that need a look I can feed them in to one and send it back to the universal patcher project.

yes they are in the offical build.

[antus]

Ok, I used AI to check the work and update the XMLs. There is a pull request to get the fixed files in UP here: https://github.com/joukoy/UniversalPatcher/pull/13

There was also noticed some other issues to do with import and export, so I will send a seperate pull request to fix those.

[antus]

Fixes for XDF import and export stride handling sent to upstream here: https://github.com/joukoy/UniversalPatcher/pull/14

Details learnt of the stride handling below:

How TunerPro XDF table strides really work (and how UniversalPatcher handles them)

TunerPro has two stride fields on a table's embedded data, Major Stride and Minor Stride. They let a definition describe tables that aren't packed tightly, such as rows with a pad byte on the end, or interleaved data. TunerPro doesn't document how they work. The rules below come from disassembling TunerPro RT (the cell address routine in TunerPro.exe). They were checked by porting that routine and comparing it with UniversalPatcher on more than 140,000 cell positions.

The basics

• A table is stored as a series of lines. A line is a row, or a column if the table is column-major (type flag 4).
• Major stride is the step from one element to the next within a line. 0 means "element size", so the data is packed.
• Minor stride controls the jump from the end of one line to the start of the next. 0 means no gap.
• In the XDF, both are in bits (mmedmajorstridebits, mmedminorstridebits). In the XDF editor and in UniversalPatcher's XML they are in bytes.

The quirk that catches everyone: what minor stride measures

Minor stride is not the size of the gap, and it is not the row length. It is the distance from the start of the last element in a line to the start of the first element of the next line.

So:

• Minor stride = element size means no gap, exactly the same as 0.
• To skip N pad bytes after each row, set minor stride to N + element size.

Example: a GM BlackBox spark table, 8-bit, 19 columns x 16 rows, with one pad byte at the end of every row. The XDF has mmedminorstridebits="16", which is 2 bytes: 1 for the element plus 1 pad byte.

Code: Select all

offset:  0    1    2   ...  18   19   20   21  ...
data:   r0c0 r0c1 r0c2 ... r0c18 PAD r1c0 r1c1 ...
                            |--------->|
              minor stride = 2 bytes, from the start of r0c18 to the start of r1c0

If the stride is lost, for example because the definition is read by something that ignores it, every row is read one byte too early. The pad byte then shows up inside the table and walks one column to the right on each row. You get a diagonal stripe of one repeated value through the table. If you see that pattern, the row stride is wrong.

The exact rule

For element size E, major stride M (0 = E), minor stride N, L elements per line:

Code: Select all

cell offset = line x ((L - 1) x |M| + E + gap) + position_in_line x |M|

gap = |N - E|,  except when |N| < |N - E|, in which case gap = |N|

For normal positive values (N >= E), the distance from one line start to the next is simply (L - 1) x M + N.

TunerPro works in bits and rounds each cell down to a whole byte at the end.

The other quirks

• Negative major stride reverses the order of elements within each line. The step is |M|.
• Negative minor stride reverses the order of the lines. Note that the gap is then |N|, not |N| - E.
• Major stride smaller than the element size is used literally, so elements overlap. TunerPro doesn't stop you.
• Minor stride between 0 and the element size gives a small, odd gap (the smaller of N and E-N). It is almost certainly a mistake in the definition.
• The XDF editor resets strides: a stride equal to the element size is reset to 0. When you change the element size, any stride smaller than the new size is reset to 0. Hand-edited XDFs don't go through this, so they can contain values the editor would never produce.
• Type flags in mmedtypeflags: 0x01 signed, 0x02 little-endian (LSB first), 0x04 column-major, 0x10000 floating point. No 0x02 flag means big-endian. This is fixed, because TunerPro only runs on x86, so "platform byte order" never comes into it.

Axes

Axis data uses the same routine, but TunerPro reads an axis as one line with indexcount values:

• Major stride is the step between axis values. A negative value reverses the axis over its full indexcount.
• The column-major flag has no effect on an axis.
• Minor stride has no effect, except that a negative minor stride makes TunerPro read the whole axis from one line before the axis address, giving garbage values. Don't do it.
• New axes in TunerPro carry mmedmajorstridebits="-32" by default. That's harmless while the axis only has labels. If you give such an axis an address in a hand-edited XDF, check that value: as written it means "reversed, 4 bytes apart".

UniversalPatcher

• Tuner XML tables use <MajorStride> and <MinorStride> in bytes, with exactly the TunerPro meaning above. XDF import converts bits to bytes.
• Older XMLs used <RowStride> (the full row-to-row distance) and <ElementStride>. These were replaced in 0.23.07. UniversalPatcher silently ignores elements it doesn't recognise, so old definitions load without any error but show the diagonal pattern above. To convert (RowStride was only applied to row-major tables):
  • MajorStride = ElementStride
  • MinorStride = RowStride - (Columns - 1) x ElementStride (with ElementStride = element size if it wasn't set)
  For the BlackBox spark table: 20 - 18 x 1 = 2. The BlackBox definitions have been updated this way (PR #13).
• A follow-up change makes UniversalPatcher match TunerPro's rule exactly everywhere it reads table cells. That covers the table editor, the value tooltip, patch creation, hex diff and table compare. It also adds negative strides, axis strides on XDF import, and a fix for the column-major flag on XDF export. The only deliberate difference: a negative minor stride on an axis is ignored, with a warning, instead of reading data from before the axis.

[sparepartsblazer]

Fixes for XDF import and export stride handling sent to upstream here: https://github.com/joukoy/UniversalPatcher/pull/14

Details learnt of the stride handling below:

How TunerPro XDF table strides really work (and how UniversalPatcher handles them)

TunerPro has two stride fields on a table's embedded data, Major Stride and Minor Stride. They let a definition describe tables that aren't packed tightly, such as rows with a pad byte on the end, or interleaved data. TunerPro doesn't document how they work. The rules below come from disassembling TunerPro RT (the cell address routine in TunerPro.exe). They were checked by porting that routine and comparing it with UniversalPatcher on more than 140,000 cell positions.

The basics

• A table is stored as a series of lines. A line is a row, or a column if the table is column-major (type flag 4).
• Major stride is the step from one element to the next within a line. 0 means "element size", so the data is packed.
• Minor stride controls the jump from the end of one line to the start of the next. 0 means no gap.
• In the XDF, both are in bits (mmedmajorstridebits, mmedminorstridebits). In the XDF editor and in UniversalPatcher's XML they are in bytes.

The quirk that catches everyone: what minor stride measures

Minor stride is not the size of the gap, and it is not the row length. It is the distance from the start of the last element in a line to the start of the first element of the next line.

So:

• Minor stride = element size means no gap, exactly the same as 0.
• To skip N pad bytes after each row, set minor stride to N + element size.

Example: a GM BlackBox spark table, 8-bit, 19 columns x 16 rows, with one pad byte at the end of every row. The XDF has mmedminorstridebits="16", which is 2 bytes: 1 for the element plus 1 pad byte.

Code: Select all

offset:  0    1    2   ...  18   19   20   21  ...
data:   r0c0 r0c1 r0c2 ... r0c18 PAD r1c0 r1c1 ...
                            |--------->|
              minor stride = 2 bytes, from the start of r0c18 to the start of r1c0

If the stride is lost, for example because the definition is read by something that ignores it, every row is read one byte too early. The pad byte then shows up inside the table and walks one column to the right on each row. You get a diagonal stripe of one repeated value through the table. If you see that pattern, the row stride is wrong.

The exact rule

For element size E, major stride M (0 = E), minor stride N, L elements per line:

Code: Select all

cell offset = line x ((L - 1) x |M| + E + gap) + position_in_line x |M|

gap = |N - E|,  except when |N| < |N - E|, in which case gap = |N|

For normal positive values (N >= E), the distance from one line start to the next is simply (L - 1) x M + N.

TunerPro works in bits and rounds each cell down to a whole byte at the end.

The other quirks

• Negative major stride reverses the order of elements within each line. The step is |M|.
• Negative minor stride reverses the order of the lines. Note that the gap is then |N|, not |N| - E.
• Major stride smaller than the element size is used literally, so elements overlap. TunerPro doesn't stop you.
• Minor stride between 0 and the element size gives a small, odd gap (the smaller of N and E-N). It is almost certainly a mistake in the definition.
• The XDF editor resets strides: a stride equal to the element size is reset to 0. When you change the element size, any stride smaller than the new size is reset to 0. Hand-edited XDFs don't go through this, so they can contain values the editor would never produce.
• Type flags in mmedtypeflags: 0x01 signed, 0x02 little-endian (LSB first), 0x04 column-major, 0x10000 floating point. No 0x02 flag means big-endian. This is fixed, because TunerPro only runs on x86, so "platform byte order" never comes into it.

Axes

Axis data uses the same routine, but TunerPro reads an axis as one line with indexcount values:

• Major stride is the step between axis values. A negative value reverses the axis over its full indexcount.
• The column-major flag has no effect on an axis.
• Minor stride has no effect, except that a negative minor stride makes TunerPro read the whole axis from one line before the axis address, giving garbage values. Don't do it.
• New axes in TunerPro carry mmedmajorstridebits="-32" by default. That's harmless while the axis only has labels. If you give such an axis an address in a hand-edited XDF, check that value: as written it means "reversed, 4 bytes apart".

UniversalPatcher

• Tuner XML tables use <MajorStride> and <MinorStride> in bytes, with exactly the TunerPro meaning above. XDF import converts bits to bytes.
• Older XMLs used <RowStride> (the full row-to-row distance) and <ElementStride>. These were replaced in 0.23.07. UniversalPatcher silently ignores elements it doesn't recognise, so old definitions load without any error but show the diagonal pattern above. To convert (RowStride was only applied to row-major tables):
  • MajorStride = ElementStride
  • MinorStride = RowStride - (Columns - 1) x ElementStride (with ElementStride = element size if it wasn't set)
  For the BlackBox spark table: 20 - 18 x 1 = 2. The BlackBox definitions have been updated this way (PR #13).
• A follow-up change makes UniversalPatcher match TunerPro's rule exactly everywhere it reads table cells. That covers the table editor, the value tooltip, patch creation, hex diff and table compare. It also adds negative strides, axis strides on XDF import, and a fix for the column-major flag on XDF export. The only deliberate difference: a negative minor stride on an axis is ignored, with a warning, instead of reading data from before the axis.

Thats a great explanation. I was wondering how it worked. Still over my head but I get the basics.

I went back and looked at my files, there is 6 spark advance tables, and 2 spark retard tables that will need MinorStride added to