Performance of $push to update large document

Working with Data
1

Good morning community!

I and My team try to make a snapshot of the rules, which shows below.
image

As you can see there is a long long array of rules (sorry for weird depth) and it is revealed that the performance of updateOne with $push operator is too slow as document size grows.

$push $each for 500 data(around 50B for each)

	//    dbAccessor_performance_test.go:35: docSize :  234.0 kB
	//    dbAccessor_performance_test.go:50: update time for 500 chunck  34.290899ms
	//
	//    dbAccessor_performance_test.go:35: docSize :  9.5 MB
	//    dbAccessor_performance_test.go:50: update time for 500 chunck  120.90932ms
	//
	//    dbAccessor_performance_test.go:35: docSize :  14.6 MB
	//    dbAccessor_performance_test.go:50: update time for 500 chunck  188.734987ms

$push for 1 data(around 50B for each)

	//	  dbAccessor_performance_test.go:72: docSize :  68.2 kB
	//	  dbAccessor_performance_test.go:85: update time for 1 doc  3.349114ms
	//
	//	  dbAccessor_performance_test.go:79: docSize :  9.1 MB
	//    dbAccessor_performance_test.go:92: update time for 1 doc  86.496973ms
	//
	//    dbAccessor_performance_test.go:72: docSize :  13.3 MB
	//    dbAccessor_performance_test.go:85: update time for 1 doc  129.689516ms

$pull

	//    dbAccessor_performance_test.go:109: docSize before delete :  4.0 MB
	//    dbAccessor_performance_test.go:115: pull tile 27.32895ms
	//    dbAccessor_performance_test.go:117: docSize after delete :  68.1 kB
	//
	//    dbAccessor_performance_test.go:109: docSize before delete :  9.9 MB
	//    dbAccessor_performance_test.go:115: pull tile 60.939563ms
	//    dbAccessor_performance_test.go:117: docSize after delete :  68.1 kB
	//
	//    dbAccessor_performance_test.go:71: docSize before delete :  14.7 MB
	//    dbAccessor_performance_test.go:77: pull tile 91.18215ms
	//    dbAccessor_performance_test.go:79: docSize after delete :  68.1 kB

I tried $push with $each and its performance is quite acceptable. However, I wonder if there is any customizable point to improve the performance of $push one element to a big array.

2

Hi @Damon_Lee ,

There is a general performance panelty with larfe arrays and it is considered an antipattern:

What we usually strongly recommend is to refactor the schema design to improve array operations like $push.

Are you familiar with our outlier pattern which should allow you to optimise cases when large array needs to be avoided

Thanks
Pavel

3

Thanks!

It seems better to use the outlier pattern with a much smaller array length.

I’ve never been heard “Massive Arrays” and also thanks to the article you shared!

1 Like
4

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.