Decoding UTF-8 Hex Strings in Python

When reverse engineering binaries, you often encounter raw hex bytes in .rodata that represent Unicode strings. Binary Ninja and similar tools sometimes render these as dots or garbage rather than the intended characters. This script bridges that gap.

Source code can be found in my Github repo / tools

The Code

import sys
def decode_hex(hex_input: str) -> str:
    hex_clean = hex_input.replace(" ", "").replace("-", "").replace("\n", "")
    raw = bytes.fromhex(hex_clean)
    return raw.decode("utf-8")
if __name__ == "__main__":
    if len(sys.argv) > 1:
        print(decode_hex(" ".join(sys.argv[1:])))
    else:
        for line in sys.stdin:
            line = line.strip()
            if line:
                try:
                    print(decode_hex(line))
                except (ValueError, UnicodeDecodeError) as e:
                    print(f"Error: {e}", file=sys.stderr)

Step 1: Imports

import sys

sys gives access to sys.argv (command-line arguments) and sys.stdin (piped input). No third-party dependencies are needed.

Step 2: The decode_hex function

def decode_hex(hex_input: str) -> str:
    hex_clean = hex_input.replace(" ", "").replace("-", "").replace("\n", "")
    raw = bytes.fromhex(hex_clean)
    return raw.decode("utf-8")

Normalisation. bytes.fromhex requires a clean hex string with no separators. The three replace calls strip spaces (Binary Ninja copy format), dashes (Wireshark/hex dump format), and stray newlines. The result is a contiguous hex string like "e2a081".

bytes.fromhex. This built-in converts a hex string directly to a bytes object. Each pair of hex digits becomes one byte. "e2a081" becomes b'\xe2\xa0\x81'.

.decode("utf-8"). UTF-8 is a variable-width encoding. Single ASCII characters occupy one byte; characters in the range U+0080 to U+07FF use two bytes; U+0800 to U+FFFF (which covers the Braille Patterns block, U+2800 to U+28FF) use three bytes. The three bytes e2 a0 81 decode to U+2801, which is ⠁. Python's str.decode handles the multi-byte sequencing automatically.

Step 3: Entry point and argument handling

if __name__ == "__main__":
    if len(sys.argv) > 1:
        print(decode_hex(" ".join(sys.argv[1:])))

When arguments are passed on the command line, sys.argv[1:] collects everything after the script name. " ".join(...) reconstructs the hex string if the shell split it across multiple tokens (e.g. python decode_utf8.py e2 a0 81 rather than quoting it). The joined string is passed directly to decode_hex.

Step 4: Stdin fallback

else:
        for line in sys.stdin:
            line = line.strip()
            if line:
                try:
                    print(decode_hex(line))
                except (ValueError, UnicodeDecodeError) as e:
                    print(f"Error: {e}", file=sys.stderr)

When no arguments are supplied, the script reads from stdin line by line, which enables piping from other tools. line.strip() removes trailing whitespace and newlines before passing to decode_hex. Two exceptions are caught explicitly:

ValueError is raised by bytes.fromhex if the input contains non-hex characters or an odd number of digits. UnicodeDecodeError is raised by .decode("utf-8") if the byte sequence is not valid UTF-8. Both are reported to stderr so they do not contaminate piped output.

Thank you for reading it

References

UTF-8 encoding specification. The canonical description of UTF-8’s variable-width byte layout, including the three-byte sequences used by the Braille Patterns block, is in the IETF standard: Yergeau, F. (2003) UTF-8, a transformation format of ISO 10646. RFC 3629. Internet Engineering Task Force. Available at: https://www.rfc-editor.org/rfc/rfc3629

Unicode Braille Patterns block. The code point assignments for U+2800 to U+28FF are defined in: The Unicode Consortium (2024) The Unicode Standard, Version 16.0, Chapter 22: Symbols. Available at: https://www.unicode.org/versions/Unicode16.0.0/

bytes.fromhex. Python Software Foundation (2024) Built-in Types: bytes.fromhex. Python 3 Documentation. Available at: https://docs.python.org/3/library/stdtypes.html#bytes.fromhex

bytes.decode. Python Software Foundation (2024) Built-in Types: bytes.decode. Python 3 Documentation. Available at: https://docs.python.org/3/library/stdtypes.html#bytes.decode

sys.argv and sys.stdin. Python Software Foundation (2024) sys -- System-specific parameters and functions. Python 3 Documentation. Available at: https://docs.python.org/3/library/sys.html

UnicodeDecodeError. Python Software Foundation (2024) Built-in Exceptions: UnicodeDecodeError. Python 3 Documentation. Available at: https://docs.python.org/3/library/exceptions.html#UnicodeDecodeError