Bin Decoder (xdf file creater and more)

Disassembly, Reassembly, Tools and devleopment. Going deep with Hardware and Software.
DWS
Posts: 129
Joined: Tue Oct 12, 2021 10:04 am
cars: Tons of Toyotas, 2003 cavi derby car, ford trucks, etc.
Location: USA
Contact:

Bin Decoder (xdf file creater and more)

Post by DWS »

I've been poking at a concept I've thought of and figured I'd make a post to log the status of my work. The very basic concept if a program that reads a bin file and spits out an xdf file. Since it's doing the reverse engineering as an automated process it works on masks to identify sections of code where sensors are read, which memory addresses they are stored in, when they are assigned to global vars, etc. Once a sensor value is tracked to a function, I count the whole function as "validated" and other references within the function can be extracted via the mask setup. In theory I could take this concept a step farther and confirm function references, validate if a given function changes between oses, etc. Why reverse engineer the same basic file 20 times because of small changes?

Anyway, I made a really basic proof of concept in c# and tested it against the 3 P04 bins I have (3100, 3400, and a 3800 SC one). The code to read the ECT sensor is the same across all 3. The function where the temp var is moved into the global var changed 3100 vs 3400/3800 (slight change but it breaks the mask). Then the 3rd layer is where the global var is used in an if statement against the fan 1 temp setting in the tune and changes so bits on a ram value which I suspect is some sort of fan status var. I'm doing my proof of concept against the known values in Universal Patcher for the time being to flesh out this concept.

I've ported the C# code to php so the concept changed slightly into a website that offers the service free. Very accessible for everyone, less bug issues, and I can track where values are found in different oses to build a database up of xdf like details plus sensor addresses and such.

I'm not sure if any other software uses this base idea or not, but I'm guessing universal patcher does to an extent since it mentions a "seek table" when it's loading things. I tried to use just a basic mask of the memory reference but quickly found a section of code in one of the bins that had conflicts, it had the reference, but it was never used against the sensor (dead code from updates?). That's why I went directly to where the sensor is read in the code and worked back to make sure the given function is actually in use.

Anyway, I figured I'd post here to see what people thought of the idea. I don't want to steal other people's work and I'll have to ask universal patcher's dev if I'm ok to use effectively their work to build this concept out. I'm not simply copying the details so I don't think it should be a problem.

I could also use some more bin files for P04 (blue + clear 80 pin connectors, v6 fwd common around 98-2005). I've been told there's something like 300 os's for these. I'd like to get some really early and some really late versions to follow the changes through the updates better and see how much matches across versions. I suspect this should work well, but maybe I'm underestimating how much GM changed the code.

Always interested in ideas and suggestions. The raw address info, sensor addresses, etc I plan to give access to as well so maybe my work can help kick start someone else on a bin that's not well known. I haven't dug into it, but for some reason P05 was having some hits match my masks for P04. There might be code reused across pcm types too.
Ford EEC-V Bin Converter (bank swapping and padding): viewtopic.php?f=41&t=8342
kur4o
Posts: 948
Joined: Sun Apr 10, 2016 9:20 pm

Re: Bin Decoder (xdf file creater and more)

Post by kur4o »

I will give some more indepth how Universal Patcher use searching for finding tables and other stuff in v6 fwd application.

It uses 3 types of searches.

Hardcoded search. You can see the results in GM-v6 info tab
It is limited to ve, maf, p/n, and checksum address, along with some 3d tables.
The format is address:rows for the table.

With that tab you can easily spot similar or identical calibrations with different p/ns.

The other type of search is tablesearch. Good for finding indepth data as ram addresses, rows, addresses, and some complex search strings with variable length combinations.
You can see the attached file for configuration. It is good for mass finding 3d tables. It is hard to use the results and to tell what is what in the results, unless you have some template.

To overcome the limitations of tablesearch we added tableseek.
With Tableseek you can find specific address that is preconfigured for use with the program. You need to supply the search string and when found the result is loaded to tuner ready for editing. Dtcs search uses similar strategy but is configured separately.

I also attached an empty tableseek templates for the most popular tables of v6s. You just need to provide the search strings.
Hint:
If you have different string for the same table, due to later code uses different sequence, you can make duplicate entry with * separator in the table name.

So you can make 3 entries in the list with different search string and even you can change other parameters in case table size was changed within calibration

Maf*98-00
Maf*01-04
Maf*05-

To get a better idea about the search strings you can load an ls1 file and open settings->tableseek
The search is really powerfull and can even search indexed tables in numerous nodes.
It can search address[taken from a found value] or absolute address[the address where the strings starts in the file].

Also you can add search within a range, use search hit[in case there is more than one entry found], search within a segment, add validation string from the found variable[used on e38].


You can also look at the upper file info tab.
You can set what gets searched upon opening a bin and also test a custom search string for numerous bins.

Save the files from attachment to xml folder.
Attachments
v6-seek_search_tables.rar
(19.63 KiB) Downloaded 284 times
kur4o
Posts: 948
Joined: Sun Apr 10, 2016 9:20 pm

Re: Bin Decoder (xdf file creater and more)

Post by kur4o »

Here are some lists I have gather so far.

From the V6_COMPETE LIST.csv you can easily identify identical calibration[all fields match except p/n][Different p/ns that can share the same definition] and also similar ones[most field matches]
Attachments
P04osidlist.txt
(14.35 KiB) Downloaded 220 times
V6_COMPETE LIST.csv
(696.6 KiB) Downloaded 238 times
GM-V6-full.csv
(19.85 KiB) Downloaded 222 times
DWS
Posts: 129
Joined: Tue Oct 12, 2021 10:04 am
cars: Tons of Toyotas, 2003 cavi derby car, ford trucks, etc.
Location: USA
Contact:

Re: Bin Decoder (xdf file creater and more)

Post by DWS »

Well that's convenient you saw my thread xD.

I don't see a GM v6 info tab in the program or in tuner. Maybe I have an older version or something?

I haven't quite got into tables and 3d tables for how they are defined, or how they are referenced/discovered. I see some lists/arrays are referenced directly to the address, maybe the compiler changes the varname[1] to the direct address. I haven't worked with assembly a whole bunch but been a programmer on and off since I was 10 (not claming I'm super amazing or anything, I try to stay very humble, I know for a fact there's a LOT more people better than me).

I saw in the logs that an xml file was referenced so I load it up and there's a table seek file and segment seek file, but they don't exist in my program directory unless windows is just really bad at finding the files lol.

I'll have to check out the files you sent, I take it you have no problem with me using this data then xD.

I like how simple UP is on the load a bin and start editing front. After I came up with the idea of masks, I saw the table seek mention in the logs so it seems like the base concept I have is more or less the same thing for universal patcher. I didn't see a "dev" mode or advanced for using the software to support looking for addresses, settings, tables, etc. Like the ECT sensor isn't needed in a tune, but figuring out other tunable settings it's pretty critical to know. That's kind of what put me down this trail. First setting I targeted was fan 1 based on ECT since that value is going to be used a lot.

Depending how your seek/mask system works, maybe I can tweak my code to match the same design you use and be able to more easily convert anything I discover into UP which gives the option to expert as xdf (why reinvent the wheel).

Ok.. I can't see out of one eye and blind in the other... File -> Mode -> Advanced.

Also, I have a small bug with UP, mainly an annoyance/gui bug. The auto resizing is the wrong height on my windows 7 setup for several windows, I always have to adjust them taller to see everything.

https://i.gyazo.com/848086d7541df8b584b ... 1d6f11.png

But this tab is just fine, weird.

https://i.gyazo.com/b58e3752e2e66f395c3 ... c08ae0.png

For tuner I'm pretty sure every value I go to edit does this. Looks like it's always about a half of a data cell too short.

https://i.gyazo.com/ad0e7918f5e5ce93d38 ... 5f9a70.png

Should I post an issue with these details on github? I'm on v 0.18.62.0

I'm not sure how much the bins change overall, but I'd think the known values in one should loosely be a list to search for in other os's for the same hardware type. I think it would be neat to know which settings are missing (for the dev side of things). It's hard to know if a bin has everything (or mostly everything) or not. Maybe I'm missing that section though or there's something simple to reference.

I swear there was a table I was looking up in the 3100 bin that wasn't found in the 3400 bin, but now it seems both are there xD.

Ultimately I'm on a learning curve poking at known bins so I can try to apply that to the P11 bin. Looks like there's zero support for this PCM so far. If I remember right I have 3 factory dumps I could pass over if that helps anything, I read them via BDM, 2 have a watch dog that currently prevents me from writing (reset has to be held low to keep the BDM active and disable watchdog, but reset can't be low to flash).

Anyway, I'll have to check those files out and play around in the UP advanced mode and see what all can be worked out using it.
Ford EEC-V Bin Converter (bank swapping and padding): viewtopic.php?f=41&t=8342
kur4o
Posts: 948
Joined: Sun Apr 10, 2016 9:20 pm

Re: Bin Decoder (xdf file creater and more)

Post by kur4o »

That gui problems seem annoying.

Some bugs have been clean up recently. Current version is 18.66

You will need the full zip package since some stuff have been reorganized.https://github.com/joukoy/UniversalPatc ... r-Full.Zip

Unzip and you can look for the xml folder where most important files are stored.

I think that later pcm might be completely different in terms of working and code. I am not sure you can transfer readily anything out of the box.

Usually the v6s you open don`t use searching yet, there is a dabase file being used from other free software called tinytuner. So the database is being loaded when you open a v6 file.
DWS
Posts: 129
Joined: Tue Oct 12, 2021 10:04 am
cars: Tons of Toyotas, 2003 cavi derby car, ford trucks, etc.
Location: USA
Contact:

Re: Bin Decoder (xdf file creater and more)

Post by DWS »

Dang you guys update too fast xD. The last one I downloaded 7 days ago. Updated and the main program gui bugs seem fixed. The popup window in tuner still has the bug.

https://i.gyazo.com/9d7831d457947cd7f5b ... 8c9ac6.png

Another fun gui thing, leaving Tuner in it's default on screen loaded spot if I open a large table the popped up window seems to low a bit too low. The overall window size is within my resolution just pushed down 50-300 pixels (the value seems to change). Not a huge deal or anything. Might just be window's default window location doing it's thing. Now that I opened and closed the window several times I see the clear down and to the right pattern, so I guess technically it can be blamed on windows lol.

I'm not seeing the full list of settings now for the P04 bins. Just these few, the last version had 100's found. Maybe I'm looking in the wrong spots. I also noticed the settings I set from the last version didn't translate to the new version, probably not a super easy thing to do though with updates etc. Also worth a note that I have "All" selected, but I poked around a bit in the icons and it's just the same list filtered down more. Personally not a fan of icons vs word labels, or atleast hover tool tips that says what the icons mean but I'm not twisting anyone's arm over little details like that.

https://i.gyazo.com/67b11ad65c627a431e0 ... 3ad8cb.png

For the P11 stuff, I'm pretty sure those are rewrote from the ground up, nothing seems to match up in past pcm models. My main goal is disable security, but tuning out limp mode and such is also on the plate. The stuff I do is for the derby community, so they want the engine to run until it blows up, no engine protections.

I've done a little java bytecode work in the past and a tiny touch of looking at assembly for pc programs, so I have a tiny bit of experience but almost none when it comes to the hardware side of the program. I'm trying to get over the learning curve so hopefully I can help with the P11 settings and such. First goal would be to identify the different sensor addresses and work my way out from there. I saw in the last version of UP that subroutines were defined, but it didn't' look like memory addresses, guessing that's some sort of OBD2 reference/name. The cpu instruction set is the same for P11, E40, and P05. I suspect what can be figured out on any of those 3 would loosely translate to all 3 of them.

Anyway, I have a basic idea of what I'm looking for now so going to hop over in Ghidra and poke at a P11 bin for a bit and try to find the ECT sensor.
Ford EEC-V Bin Converter (bank swapping and padding): viewtopic.php?f=41&t=8342
kur4o
Posts: 948
Joined: Sun Apr 10, 2016 9:20 pm

Re: Bin Decoder (xdf file creater and more)

Post by kur4o »

About the gui bug on tables.

Can you check if the autoresize is checked. when you open a table click on settings-> autoresize
DWS
Posts: 129
Joined: Tue Oct 12, 2021 10:04 am
cars: Tons of Toyotas, 2003 cavi derby car, ford trucks, etc.
Location: USA
Contact:

Re: Bin Decoder (xdf file creater and more)

Post by DWS »

Guessing remember window size might be the setting you're looking for?

Image
Ford EEC-V Bin Converter (bank swapping and padding): viewtopic.php?f=41&t=8342
kur4o
Posts: 948
Joined: Sun Apr 10, 2016 9:20 pm

Re: Bin Decoder (xdf file creater and more)

Post by kur4o »

Open a random table. There is another setting tab there. The autoresize is there.
DWS
Posts: 129
Joined: Tue Oct 12, 2021 10:04 am
cars: Tons of Toyotas, 2003 cavi derby car, ford trucks, etc.
Location: USA
Contact:

Re: Bin Decoder (xdf file creater and more)

Post by DWS »

Ahh i see. It is checked.

Image
Ford EEC-V Bin Converter (bank swapping and padding): viewtopic.php?f=41&t=8342
Post Reply