OBDX Development - Developer Tools and Suggestions

Programs / Tools / Scripts
User avatar
Tazzi
Posts: 3422
Joined: Thu May 17, 2012 8:53 pm
cars: VE SS Ute
Location: WA
Contact:

Re: OBDX Development - Developer Tools and Suggestions

Post by Tazzi »

antus wrote:Do you find yourself doing lots of bit level manipulation? There are many sheets of 'bit hacks' out there which are great to optimize code in small embedded applications like this https://cheatography.com/jsondhof/cheat ... bit-hacks/ https://graphics.stanford.edu/~seander/bithacks.html
You might find you can save some bytes using smaller data types too - depending your coding style if your using 32 or 64 bit data bytes in places where you know the range would fit in a single byte (signed or unsigned) you can shrink the ram usage and the code that does the work with those variables by using smaller data types. You have to be careful though, its easy to get too excited and shrink something too far, and it can be difficult to find what you did. Make sure you use git for your code or something, so that you can divide and conquer if something like that comes up by running different firmware revisions to see where it was introduced and what you changed at that time.
Surprisingly we are ok for RAM, its just flash space that is at its capacity.
I have honestly gone through every line of code to check for variable type sizes, and removed any unused functions. Its just the shear nature of bulk coding to support so many commands for both ELM and DVI sets.

Something thats a bit experimental but seems promising, is I have been reading about libraries can be 'precompiled' and used in projects to help save compiling time, or to ensure end users do not mix/match incorrect libraries.
What this means to me, is I could compile a library using the highest optimization settings, and then use that in the current project that does not have optimizations enabled.
If the ELM,DVI and protocol libraries could be compressed with optimization, then this would save many thousands of bytes without jeopardizing stability.

On other news, OBDXplorer has now been released. Theres a couple fixes already in the works:
1) Seems some laptops with bluetooth are hanging during the auto connect stage. Not too sure why this is occurring since it does not attempt to connect to any comport, only check the 'name' of the port so it only finds OBDX Pro devices. I will need to link something via bluetooth to produce a comport and see if that results in the same hanging.

2) Some PIDs may have the wrong size allocated. This results in logging setup procedure failing as the PCM recognizes the wrong size for the PID is used. This will require going through the supported list of PIDs and testing them out to see which fail to start.
Your Local Aussie Reverse Engineer
Contact for Software/Hardware development and Reverse Engineering
Site:https://www.envyouscustoms.com
Mob:+61406 140 726
Image
User avatar
Tazzi
Posts: 3422
Joined: Thu May 17, 2012 8:53 pm
cars: VE SS Ute
Location: WA
Contact:

Re: OBDX Development - Developer Tools and Suggestions

Post by Tazzi »

Weird.. I can't replicate this hanging comport issue... looks like Im going to have to add a shitload of debug messages and a default timeout option so that users can get out of the screen.
The ports are completely ignored since their 'name' does not contain OBDX Pro. Even when paired with the OBDX Pro tool, it still shows only com3/4 so it ignores it.

I know its a bluetooth issue since the people affected turn off bluetooth on their laptop and it proceeds instantly. I wonder if certain windows versions try to auto connect to a bluetooth device when requesting details about the comport, even though these details are available while disconnected?

A bit of a hack method is to open a little message box asking if they accept obdxplorer to disable bluetooth when it detects that its on. Its not ideal but I can't seem to replicate the problem to diagnose.
Your Local Aussie Reverse Engineer
Contact for Software/Hardware development and Reverse Engineering
Site:https://www.envyouscustoms.com
Mob:+61406 140 726
Image
User avatar
antus
Site Admin
Posts: 8237
Joined: Sat Feb 28, 2009 8:34 pm
cars: TX Gemini 2L Twincam
TX Gemini SR20 18psi
Datsun 1200 Ute
Subaru Blitzen '06 EZ30 4th gen, 3.0R Spec B
Contact:

Re: OBDX Development - Developer Tools and Suggestions

Post by antus »

The variable sizes dont just save ram, they also save program space. For example, in x86 asm, a 32 bit move of a value in to a register takes 5 bytes where 1 byte is opcode, 4 bytes are the 32 bit value, including the high bytes(s) even if they're zero.

Code: Select all

BE 9A 77 01 00          mov     esi, 1779Ah
Whereas a 16 bit move in to a register is 3 bytes, as the high bytes are not required in program space. And another byte saved again if you use a byte data type. Thus by making sure you get a variable right (in ram) you save space many times over in the code, too.

Code: Select all

B8 0A 00                                mov     ax, 0Ah
Whats more, the compiler will size the data value either by your syntax or by the data type of the destination. So if your saving 'a' in to a 32 bit data type, the 'a' reference in the assembler will be 4 bytes.

This 'precompiling' your talking about sounds like the difference between the compiling stage and the linking stage. Normally your compiler and linker would be two different stages, you can use more advanced build system configuration to build different object/library files with different compiler configuration then when you link them the binary payload will be copied (not recompiled) in to the resulting executable. Most build systems would check the modification times on the objects and not rebuild them if the source has not changed as a compile time optimisation, so you are probably getting this behavior out of the box.
Have you read the FAQ? For lots of information and links to significant threads see here: http://pcmhacking.net/forums/viewtopic.php?f=7&t=1396
In-Tech
Posts: 779
Joined: Mon Mar 09, 2020 4:35 pm
Location: California

Re: OBDX Development - Developer Tools and Suggestions

Post by In-Tech »

I hate to butt in, it still seems to me that it will always boil back to 8 bit in its' real machine language. It sure seems like, from what I have seen, the original 8 bit opcodes still work regardless if it's 32 or 64 bit. I gotta drag out my old crapola and play but time is a problem at this moment. I loved writing in 8 bit assy since we could figure timing in opcode ticks and didn't have the overhead of the software nor 'puter issues :(
User avatar
Tazzi
Posts: 3422
Joined: Thu May 17, 2012 8:53 pm
cars: VE SS Ute
Location: WA
Contact:

Re: OBDX Development - Developer Tools and Suggestions

Post by Tazzi »

antus wrote:The variable sizes dont just save ram, they also save program space. For example, in x86 asm, a 32 bit move of a value in to a register takes 5 bytes where 1 byte is opcode, 4 bytes are the 32 bit value, including the high bytes(s) even if they're zero.

Code: Select all

BE 9A 77 01 00          mov     esi, 1779Ah
Whereas a 16 bit move in to a register is 3 bytes, as the high bytes are not required in program space. And another byte saved again if you use a byte data type. Thus by making sure you get a variable right (in ram) you save space many times over in the code, too.

Code: Select all

B8 0A 00                                mov     ax, 0Ah
Whats more, the compiler will size the data value either by your syntax or by the data type of the destination. So if your saving 'a' in to a 32 bit data type, the 'a' reference in the assembler will be 4 bytes.

This 'precompiling' your talking about sounds like the difference between the compiling stage and the linking stage. Normally your compiler and linker would be two different stages, you can use more advanced build system configuration to build different object/library files with different compiler configuration then when you link them the binary payload will be copied (not recompiled) in to the resulting executable. Most build systems would check the modification times on the objects and not rebuild them if the source has not changed as a compile time optimisation, so you are probably getting this behavior out of the box.
Went through the coding today, and only found a couple places where I could reduce from 32bit to 8bit, and a 32bit to 16bit. Other then that, Iv basically optimized data sizes as much as I can.

As for the precompiled options, thats correct, its all linked together at the final step. The main thing is I can't seem to set specific libraries/code to be left unoptimized, thus compiling the libraries separate so they can use compiler optimization settings should help reduce size drastically.
For now we are good to have have about 1.5kb free, which allows for some very minor additions/modifications if ever needed, but its no where near the size needed to add any other protocols such as kline or even FD CAN. We will have to upgrades to a bigger flash chip to be able to squeeze those next two protocols in if maintaining both ELM and DVI protocols.

moving forward though. I am just finishing off another batch of GTs to head off to Pete. We should have our first initial release of the GTs in the next week or so (fingers crossed). The driver installer has been added to the website, but I still have to compile the J2534 API installer.
I have not completed the DPDU for the GTs.. since.. well... that API still haunts my dreams... think I need to leave that a little longer before I start getting CANbus+ALDL 100% implemented for the GT to use with Tech2win.
Your Local Aussie Reverse Engineer
Contact for Software/Hardware development and Reverse Engineering
Site:https://www.envyouscustoms.com
Mob:+61406 140 726
Image
User avatar
Tazzi
Posts: 3422
Joined: Thu May 17, 2012 8:53 pm
cars: VE SS Ute
Location: WA
Contact:

Re: OBDX Development - Developer Tools and Suggestions

Post by Tazzi »

Just got canbus based ECUs now working with OBDXplorer, but we do have a hiccup.

We are processing data SO fast that its actually resulting in the graphing to lag behind. This is due to the chart trying to refresh every single time a single parameter is updated.

Now, I do have a couple solutions for this:
1) Implement a timer which will check for updates every say.. 24ms, and then add all newly received parameters to their displays/charts.
2) Look into pausing screen updates until all PIDs from a DPID frame are processed and then allow an update all at once.

The delay gets worse as time goes on, so its an absolute must to deal with it now then later.


On a completely separate topic. I programmed a vehicle in NZ using remote J2534. This required additional attention to J2534 API which needed to add some additional smarts to accommodate for lag. The biggest one being the tester present since GMs software actually manually sends this instead of automating it in the tool. With that said, it did work, but the delay caused through an international connection is not great, the process took much longer then when in person.

The only solution I can think for this is to host a windows machine/server at the location required, then remote in and use and use the J2534 software there to significantly reduce lag.
This is no way works in a commercial environment since nothing stops someone from installation malicious software to that host machine, but it does fix the lag issue.

But could be turned into a tutorial for workshops to follow? I do believe the world is moving towards remote programming for the automotive scene!
Your Local Aussie Reverse Engineer
Contact for Software/Hardware development and Reverse Engineering
Site:https://www.envyouscustoms.com
Mob:+61406 140 726
Image
User avatar
antus
Site Admin
Posts: 8237
Joined: Sat Feb 28, 2009 8:34 pm
cars: TX Gemini 2L Twincam
TX Gemini SR20 18psi
Datsun 1200 Ute
Subaru Blitzen '06 EZ30 4th gen, 3.0R Spec B
Contact:

Re: OBDX Development - Developer Tools and Suggestions

Post by antus »

I'd go for an option 3, similar to your option 2 but async and ready to drop screen updates. I think you need current pids stored in ram and your data thread overwriting the last value an streaming the data to disk if your saving the log, and do that in a background thread. Then have another thread driving the front end, and running a timer so every, say 100ms read the last values and update the gauges (like your #1). This should be enough for app users to see recent enough data, and if they need frame accurate data they can load it from disk and go frame by frame. If 100ms is too slow, you could go 50 or 20. Fast enough to look smooth, slow enough to give a laptop a chance to slow down the cpu/gpu if it can and save power. You could also make it settable as a config item, though that might be going too far. Both of these threads shouldnt be in your main thread so its free to handle clicks quickly to keep the UI fluid to use.
Have you read the FAQ? For lots of information and links to significant threads see here: http://pcmhacking.net/forums/viewtopic.php?f=7&t=1396
User avatar
Tazzi
Posts: 3422
Joined: Thu May 17, 2012 8:53 pm
cars: VE SS Ute
Location: WA
Contact:

Re: OBDX Development - Developer Tools and Suggestions

Post by Tazzi »

antus wrote:I'd go for an option 3, similar to your option 2 but async and ready to drop screen updates. I think you need current pids stored in ram and your data thread overwriting the last value an streaming the data to disk if your saving the log, and do that in a background thread. Then have another thread driving the front end, and running a timer so every, say 100ms read the last values and update the gauges (like your #1). This should be enough for app users to see recent enough data, and if they need frame accurate data they can load it from disk and go frame by frame. If 100ms is too slow, you could go 50 or 20. Fast enough to look smooth, slow enough to give a laptop a chance to slow down the cpu/gpu if it can and save power. You could also make it settable as a config item, though that might be going too far. Both of these threads shouldnt be in your main thread so its free to handle clicks quickly to keep the UI fluid to use.
Ant, you legend.
Applied the options plus background thread, and we are in action! No more delays!! woohoo!
Your Local Aussie Reverse Engineer
Contact for Software/Hardware development and Reverse Engineering
Site:https://www.envyouscustoms.com
Mob:+61406 140 726
Image
User avatar
antus
Site Admin
Posts: 8237
Joined: Sat Feb 28, 2009 8:34 pm
cars: TX Gemini 2L Twincam
TX Gemini SR20 18psi
Datsun 1200 Ute
Subaru Blitzen '06 EZ30 4th gen, 3.0R Spec B
Contact:

Re: OBDX Development - Developer Tools and Suggestions

Post by antus »

good to hear :thumbup:
Have you read the FAQ? For lots of information and links to significant threads see here: http://pcmhacking.net/forums/viewtopic.php?f=7&t=1396
User avatar
Tazzi
Posts: 3422
Joined: Thu May 17, 2012 8:53 pm
cars: VE SS Ute
Location: WA
Contact:

Re: OBDX Development - Developer Tools and Suggestions

Post by Tazzi »

New problem!
While that has now worked fantastic, we now have a problem of an unhandled exception arising, which is actually coming from the graphing GUI itself. If I disable updating the graphs, the problem disappears.

The issue is when a list/queue is altered while another thread/action is using it.
I also lock every list/queue when I am using them, which is why the issue doesn't seem to be set off even though I am still displaying to dashboard and listview.

The graphs themselves are data bound to some datapoint lists, which I also lock when using.. but appears the issue still persists.

Its almost like the internal graphing is actually using the list while Im trying to add new points to it?? Adding a try/catch doesnt seem to help either.

Going to have to add line by line and identify the exact issue.

*Edit
Possibly fixed. Appears updating the data source while in a background thread results in the graphing to have a small heart attack and hard crash.
Just ran a quick invoke to update via main UI thread, and problem solved! (At least, two minutes in and no problems!)
Your Local Aussie Reverse Engineer
Contact for Software/Hardware development and Reverse Engineering
Site:https://www.envyouscustoms.com
Mob:+61406 140 726
Image
Post Reply