Why is `bytes.Buffer` only updating offsets for reads and not for writes?

Explain
1

I am new to Odin and started writing some code to write to a buffer (file in the real case, bytes.Buffer in a test to check the result). However, I soon figured out that the behaviour is different (I am on linux).

In particular bytes.Buffer does not update it’s internal offset on writes, so if after I write I want to know the current offset, seek will return 0. This forces me to keep track of the offset myself.

Am I missing something, is it intended to not behave the same or is this unexpected behavior?

Here is the core lib code for write_buffer for easy browsing.

2

try to use len(b.buf)

3

With my below example, tracking the last offset is not required, as the buffer will grow on it’s own, but the bytes.buffer_write does return n bytes written so that the last offset can be tracked if needed.

b: bytes.Buffer
bytes.buffer_init_allocator(&b, 0, 0, context.allocator)
defer bytes.buffer_destroy(&b)
offset := bytes.buffer_write(&b, transmute([]byte)string("hellope")) or_else 0
offset += bytes.buffer_write(&b, transmute([]byte)string(" world")) or_else 0

fmt.println(bytes.buffer_to_string(&b), "length:", bytes.buffer_length(&b), "offset:", offset)
4

Thanks for the replies, I already solved the problem by tracking offsets myself (similar to your suggestion @xuul). In my code I don’t want to rely on a particular io.Writer implementation, so I can’t simply len(w.buf).

I just feel this breaks the abstraction. As in Liskov principle… I know odin is not OOP, but that holds if you make an “interface” like io.Writer, no?

5

You may have a good point, considering the Buffer type has a last_read field. It does beg the question of why does it not have a last_write field also? Maybe there is a technical reason only the creator of that library can answer?

Playing with it further, Buffer does have an off field (for offset I believe). It appears that is used for the last_read operation. From that I can only guess that requiring the user to track their own last write offset is trivial compared to tracking read offset. This all from just a quick glance.

6

For reproducibility, I made a self-enclosed pasteable example:

package bug

import "core:io"
import "core:os"
import "core:bytes"
import "core:fmt"

content: string : "Hello"

main :: proc() {
    // File
    file, _ := os.open("file.txt", {.Create, .Write})
    defer os.close(file)

    file_writer := os.to_writer(file)
    write_and_seek(file_writer, "file")

    // Buffer
    buf: bytes.Buffer
    defer bytes.buffer_destroy(&buf)

    buf_writer := bytes.buffer_to_stream(&buf)
    write_and_seek(buf_writer, "buffer")
}

write_and_seek :: proc(w: io.Write_Seeker, name: string) {
    io.write(w, transmute([]u8)content)

    n, _ := io.seek(w, 0, io.Seek_From.Current)
    fmt.printfln("%s offset after write: %v", name, n)
}

This on my pc gives:

file offset after write: 5
buffer offset after write: 0
7

First problem I noticed with the example. On subsequent runs, file is over-written and not appended to if it already exists and contains data.

If I change seek to use io.Seek_From.End, it works closure to the desired way you describe, but I still see a problem, though offset in this case is really just the length. The file.txt is appended to, but there is an overlap. On second run it should have advanced to 15, 20, but instead advances 10, 15, where 10 is an overlap. Otherwise the offset/length value is correct for the data that does exist in both cases.

content: string : "Hello"

write_and_seek :: proc(w: io.Write_Seeker, name: string) {
	io.write(w, transmute([]u8)content)
	n := io.seek(w, 0, io.Seek_From.End) or_else 0
	fmt.printfln("%s offset after write: %v", name, n)
}

main :: proc() {
	file, _ := os.open("file.txt", {.Create, .Write})
	defer os.close(file)

	file_writer := os.to_writer(file)
	write_and_seek(file_writer, "file1")
	write_and_seek(file_writer, "file2")

	// Buffer
	buf: bytes.Buffer
	defer bytes.buffer_destroy(&buf)

	buf_writer := bytes.buffer_to_stream(&buf)
	write_and_seek(buf_writer, "buffer1")
	write_and_seek(buf_writer, "buffer2")

	fmt.printfln("%#w", string(buf.buf[:]))
}

Ouput

file1 offset after write: 5
file2 offset after write: 10
buffer1 offset after write: 5
buffer2 offset after write: 10

Output ran a second time, with the file.txt already containing data from first. Buffer of course is reinitialized, so is the same…

file1 offset after write: 10
file2 offset after write: 15
buffer1 offset after write: 5
buffer2 offset after write: 10
8

Ok, I think I’m starting to get into the psychology of what seek is meant for. Seek is gonna be useful for different things depending on the goal. Not sure I understand the goal, but if it is to always append to the end of a file, be it new or existing and also always append to the end of a byte buffer, I would do the below. There’s not gonna be a silver bullet for every io.stream situation, so forming a collection of procedures may be necessary if there exists incompatible goals. If I’m wrong in my assumption on the desired outcome, please help me understand and I’ll change my example to reflect that goal.

main :: proc() {
	// File
	file, _ := os.open("file.txt", {.Create, .Write})
	defer os.close(file)

	file_writer := os.to_writer(file)
	seek_and_write(file_writer, "file1")
	seek_and_write(file_writer, "file2")
	seek_and_write(file_writer, "file3")

	// Buffer
	buf: bytes.Buffer
	defer bytes.buffer_destroy(&buf)

	buf_writer := bytes.buffer_to_stream(&buf)
	seek_and_write(buf_writer, "buffer1")
	seek_and_write(buf_writer, "buffer2")
	seek_and_write(buf_writer, "buffer3")
}

content: string : "Hello"

seek_and_write :: proc(w: io.Write_Seeker, name: string) -> (offset: i64) {
	s := io.seek(w, 0, .End) or_else 0
	n := io.write(w, transmute([]u8)content) or_else 0
	offset = s+i64(n)
	fmt.printfln("%s offset after write: %v", name, offset)
	return
}

Output, run 1 run 2:

PS C:\Users\foo\Documents\projects\odin\scratch> odin run .
file1 offset after write: 5
file2 offset after write: 10
file3 offset after write: 15
buffer1 offset after write: 5
buffer2 offset after write: 10
buffer3 offset after write: 15
PS C:\Users\foo\Documents\projects\odin\scratch> odin run .
file1 offset after write: 20
file2 offset after write: 25
file3 offset after write: 30
buffer1 offset after write: 5
buffer2 offset after write: 10
buffer3 offset after write: 15
1 Like
9

@xuul, thanks for the investigation! I haven’t thought about .End very well and indeed that fixes my issue (assuming of course we are just always appending to the end)!

I still think this is a bug for .Current in the core lib but the .End will probably be enough in my program!